I have a csv file with the following contents:
INTERB-MNT,2008-09-10T21:05:38Z,2008-09-10T21:05:38Z,MARIA
How can I use sed to replace the characters 'T' and 'Z', such that the contents of the file are changed to the following?:
INTERB-MNT,2008-09-10,21:05:38,UTC,2008-09-10,21:05:38,UTC,MARIA
I tried the following, but obviously I'm missing something because that does not produce the desired results:
sed -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}.T.[0-9]{2}:[0-9]{2}:[0-9]{2}Z/[0-9]{4}-[0-9]{2}-[0-9]{2},[0-9]{2}:[0-9]{2}:[0-9]{2}UTC/g'
To keep your text after substitution, you have to capture input with parens, and then use \1 through \9 to refer to the captured matching in the substitution part. To be able to use \1 through \9 backreferences, you have to use -E or -r options.
The command will look like this:
sed -r 's/(.+)T(.+)Z/\1,\2,UTC/g'
But this can't be used: the T will match the last part of the string because (.+) is greedy. So your idea to match 2008-09-10 and 21:05:38 pattern is good. You ended up with this:
sed -r 's/([0-9]{4}-[0-9]{2}-[0-9]{2})T([0-9]{2}:[0-9]{2}:[0-9]{2})Z/\1,\2,UTC/g'
This works. You could also use this simpler command:
sed -r 's/(....-..-..)T(..:..:..)Z/\1,\2,UTC/g'
It is easier to read and write, and a false positive is very unlikely. It depends on your needs.
Related
I have a text file with the following pattern written to it:
TIME[32.468ms] -(3)-............."TEXT I WANT TO KEEP"
I would like to discard the first part of each line containing
TIME[32.468ms] -(3)-.............
To test the regular expression I've tried the following:
cat myfile.txt | egrep "^TIME\[.*\]\s\s\-\(3\)\-\.+"
This identifies correctly the lines I want. Now, to delete the pattern I've tried:
cat myfile.txt | sed s/"^TIME\[.*\]\s\s\-\(3\)\-\.+"//
but it just seems to be doing the cat, since it shows the content of the complete file and no substitution happens.
What am I doing wrong?
OS: CentOS 7
With your shown samples, please try following grep command. Written and tested with GNU grep.
grep -oP '^TIME\[\d+\.\d+ms\]\s+-\(\d+\)-\.+\K.*' Input_file
Explanation: Adding detailed explanation for above code.
^TIME\[ ##Matching string TIME from starting of value here.
\d+\.\d+ms\] ##Matching digits(1 or more occurrences) followed by dot digits(1 or more occurrences) followed by ms ] here.
\s+-\(\d+\)-\.+ ##Matching spaces91 or more occurrences) followed by - digits(1 or more occurrences) - and 1 or more dots.
\K ##Using \K option of GNU grep to make sure previous match is found in line but don't consider it in printing, print next matched regex part only.
.* ##to match till end of the value.
2nd solution: Adding awk program here.
awk 'match($0,/^TIME\[[0-9]+\.[0-9]+ms\][[:space:]]+-\([0-9]+\)-\.+/){print substr($0,RSTART+RLENGTH)}' Input_file
Explanation: using match function of awk, to match regex ^TIME\[[0-9]+\.[0-9]+ms\][[:space:]]+-\([0-9]+\)-\.+ which will catch text which we actually want to remove from lines. Then printing rest of the text apart from matched one which is actually required by OP.
This awk using its sub() function:
awk 'sub(/^TIME[[][^]]*].*\.+/,"")' file
"TEXT I WANT TO KEEP"
If there is replacement, sub() returns true.
$ cut -d'"' -f2 file
TEXT I WANT TO KEEP
You may use:
s='TIME[32.468ms] -(3)-............."TEXT I WANT TO KEEP"'
sed -E 's/^TIME\[[^]]*].*\.+//'
"TEXT I WANT TO KEEP"
The \s regex extension may not be supported by your sed.
In BRE syntax (which is what sed speaks out of the box) you do not backslash round parentheses - doing that turns them into regex metacharacters which do not match themselves, somewhat unintuitively. Also, + is just a regular character in BRE, not a repetition operator (though you can turn it into one by similarly backslashing it: \+).
You can try adding an -E option to switch from BRE syntax to the perhaps more familiar ERE syntax, but that still won't enable Perl regex extensions, which are not part of ERE syntax, either.
sed 's/^TIME\[[^][]*\][[:space:]][[:space:]]-(3)-\.*//' myfile.txt
should work on any reasonably POSIX sed. (Notice also how the minus character does not need to be backslash-escaped, though doing so is harmless per se. Furthermore, I tightened up the regex for the square brackets, to prevent the "match anything" regex you had .* from "escaping" past the closing square bracket. In some more detail, [^][] is a negated character class which matches any character which isn't (a newline or) ] or [; they have to be specified exactly in this order to avoid ambiguity in the character class definition. Finally, notice also how the entire sed script should normally be quoted in single quotes, unless you have specific reasons to use different quoting.)
If you have sed -E or sed -r you can use + instead of * but then this complicates the overall regex, so I won't suggest that here.
A simpler one for sed:
sed 's/^[^"]*//' myfile.txt
If the "text you want to keep" always surrounded by the quote like this and only them having the quote in the line starting with "TIME...", then:
sed -n '/^TIME/p' file | awk -F'"' '{print $2}'
should get the line starting with "TIME..." and print the text within the quotes.
Thanks all, for your help.
By the end, I've found a way to make it work:
echo 'TIME[32.468ms] -(3)-.............TEXT I WANT TO KEEP' | grep TIME | sed -r 's/^TIME\[[0-9]+\.[0-9]+ms\]\s\s-\(3\)-\.+//'
More generally,
grep TIME myfile.txt | sed -r ‘s/^TIME\[[0-9]+\.[0-9]+ms\]\s\s-\(3\)-\.+//’
Cheers,
Pedro
I have a large file that contains timestamps in the following format:
2018-08-22T13:06:04.442774Z
I would like to add double quotes around all the occurrences that match this specific expression. I am trying to use sed, but I don't seem to be able to find the right command. I am trying something around these lines:
sed -e s/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}.[0-9]\{6\}Z/"$0"/g my_file.json
and I am pretty sure that the problem is around my "replace" expression.
How should I correct the command?
Thank you in advance.
You should wrap the sed replacement command with single quotes and use & instead of $0 in the RHS to replace with the whole match:
sed 's/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\.[0-9]\{6\}Z/"&"/g' file > outfile
See the online demo
Also, do not forget to escape the . char if you want to match a dot, and not any character.
You may also remove excessive escapes if you use ERE syntax:
sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{6}Z/"&"/g'
If you want to change the file inline, use the -i option,
sed -i 's/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\.[0-9]\{6\}Z/"&"/g' file
The following works:
sed 's/\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\.[0-9]\{6\}Z\)/"\1"/g' my_file.json
multiple modifications:
wrap command in single quotes
use \( and \) to create a group (referenced by '\1` in the replacement section)
escape the '.' and '{' and '}' characters
Having the following string inside of a text file.
{"_job":"delete","query":{"query":{"bool":{"must":[{"term":{"_id":"28381"}}],"should":[]}}},"script":{"inline":"ctx._source.meta='This
is a ' test string Peedr'"},"timestamp":1518165383,"host":"","port":"9200","index":"","docType":"","customIndexer":""}
I would like to replace all the ' that are inside the ctx._source.meta='' part with \' using sed.
In the example above I've This is a ' test string Peedr which I would like to convert to This is a \' test string Peedr, so the desired output would be:
{"_job":"delete","query":{"query":{"bool":{"must":[{"term":{"_id":"28381"}}],"should":[]}}},"script":{"inline":"ctx._source.meta='This
is a \' test string
Peedr'"},"timestamp":1518165383,"host":"","port":"9200","index":"","docType":"","customIndexer":""}
I'm using the following regex to get the ' that is inside the ctx._source.meta string (3rd capture group).
(meta=')(.*?)(')(.*?)(')
I've the regex, but I dont know how to use the sed comand in order to replace the 3rd capture group with \'.
Can someone give me a hand and tell me the sed comand I have to use?
Thanks in advance
sed generally does not support the Perl regex extensions, so the non-greedy .*? will probably not do what you hope. If you want to use Perl regex, use Perl!
perl -pe "s/(meta='.*?)(')(.*?')/\$1\\\\\$2\$3/"
This will still not necessarily work if the input is malformed; a better approach would be to specifically exclude single quotes from the match, and then you don't need the non-greedy matching.
sed "s/\\(meta='[^']*\\)'\\([^']*'\\)/\\1\\\\'\\2/"
In both cases, the number of backslashes required to escape the backslashes inside the shell's double quotes is staggering.
You put back-references to groups except one you want to replace. There is a better way to accomplish same task:
sed -E "s/(ctx\._source\.meta=')([^']*)(')([^']*')/\1\2\\'\4/"
You may use:
sed "s/ ' / \\\' /g" sample.txt
The first part will instruct sed to only look for a single quote between 2 spaces, as such ctx._source.meta='This and string Peedr'"} will not match, hence will not be changed.
Edit:
At the poster's request, I edited my sed command to apply to extra use cases:
sed "s/\(ctx._source.meta='.*\)'\(.*Peedr'\"\)/\1\\\'\2/g"
Sorry to have to post this as it seems rather basic, but I can't tell why the heck it's not working:
sed -r -i.bak 's/-- Database: (.*?)\n/CREATE DATABASE \1\n/g' file.sql
The file contains a line that looks like this:
-- Database: `omptest`
and doing the exact same thing as a find/replace in sublime works fine:
Find: -- Database: (.*?)\n
Replace with: CREATE DATABASE \1\n
sed doesn't seem to like the \1 unless I either escape the parentheses surrounding the capture group or use -r for expanded regex -- otherwise it throws sed: -e expression #1, char 53: invalid reference \1 on `s' command's RHS.
Regardless, when I use the -r or escape the parentheses, it doesn't edit the file, as though it hasn't worked. I have no idea why.
I think I'm either forgetting or just now discovering some weird property of sed regex but I can't for the life of me figure out what the issue might be. Help appreciated!
sed works on a line-based basis and doesn't see the newlines in a file (unless you do special things). Use
# vvv----------------vvv--- no newlines
sed -r -i.bak 's/-- Database: (.*?)/CREATE DATABASE \1/g' file.sql
Addendum: Three more things to note, perhaps:
.*? does exactly the same as .* in sed. The former can be used for non-greedy matching in some other regex engines but not in sed. The ? does not do any harm, but it doesn't do anything at all either.
Since the regex matches greedily to the end of the line, the /g flag doesn't have any impact. There's never going to be a second match because the first extends to the end of the line.
Since the capture group is put right back where it was found, it doesn't really need to be captured it in the first place.
From the first two observations, we might arrive at a simplified version of the original script:
sed -r -i.bak 's/-- Database: (.*)/CREATE DATABASE \1/' file.sql
...and because of the third, we could ditch those changes again and just use
sed -i.bak 's/-- Database: /CREATE DATABASE /' file.sql
instead.
I am trying to replace an expression using sed. The regex works in vim but not in sed. I'm replacing the last dash before the number with a slash so
/www/file-name-1
should return
/www/file-name/1
I am using the following command but it keeps outputting /www/file-name/0 instead
sed 's/-[0-9]/\/\0/g' input.txt
What am I doing wrong?
You must surround between parentheses the data to reference it later, and sed begins to count in 1. To recover all the characters matched without the need of parentheses, it is used the & symbol.
sed 's/-\([0-9]\)/\/\1/g' input.txt
That yields:
/www/file-name/1
You need to capture using parenthesis before you can back reference (which start a \1). Try sed -r 's|(.*)-|\1/|':
$ sed -r 's|(.*)-|\1/|' <<< "/www/file-name-1"
/www/file-name/1
You can use any delimiter with sed so / isn't the best choice when the substitution contains /. The -r option is for extended regexp so the parenthesis don't need to be escaped.
It seems sed under OS X starts counting backreferences at 1. Try \1 instead of \0