Linux short by symbol - regex

lets say I got this file:
https://www.wireless.miau.com/business/index.jsp?_requestid=839
https://www.wireless.miau.com/business/index.jsp?_requestid=859
https://www.wireless.miau.com/business/index.jsp?_requestid=886
https://www.wireless.miau.com/business/index.jsp?_requestid=897
https://www.wireless.miau.com/business/index.jsp?source=EENT042114500148N&_requestid=26&wtSlotClick=1-006Y3D-1-4&rel=nofollow
https://www.wireless.miau.com/business/index.jsp?token=46e134202bdd498c3369812e93c040122ecaf6b3
https://www.wireless.miau.com/business/index.jsp?token=Ae18464df3d728b2aa98286e843cd38cfa82c49d1&B2B_GTM=
https://www.wireless.miau.com/business/optin/optInEmailPopup.jsp?displayMode=1&from=businessCenter&bref=IBD903j370921100&wtLinkName=Subscribenow&wtLinkLoc=RN
https://www.wireless.miau.com/business/phones/phone_details_main.jsp?_DARGS=/business/templates/od_page_begin.jsp.pageLogoutForm
https://www.wireless.miau.com/business/policies/site_access.jsp;dsessionid=EBS51RW00NBS3B4R0ENCFEY?wtLinkName=TermsOfUse&wtLinkLoc=FTR
https://www.wireless.miau.com/business/shop/shop_cru_home.jsp?_DARGS=/business/shop/shop_cru_home.jsp.form_1
https://www.wireless.miau.com/pct/contract_acceptance/agreement.jsp?_DARGS=/pct/contract_acceptance/business_agreement_frag.jsp.pageBusinessForm
https://xdme.wireless.miau.com/jsp/login/login.jsp?Exp=Y&FP=Y
https://youachieve.miau.com/news.cfm?id=1204
https://youachieve.miau.com/news.cfm?id=1214
And I want this output:
https://www.wireless.miau.com/business/index.jsp?_requestid=839
https://www.wireless.miau.com/business/index.jsp?source=EENT042114500148N&_requestid=26&wtSlotClick=1-006Y3D-1-4&rel=nofollow
https://www.wireless.miau.com/business/index.jsp?token=46e134202bdd498c3369812e93c040122ecaf6b3
https://www.wireless.miau.com/business/index.jsp?token=Ae18464df3d728b2aa98286e843cd38cfa82c49d1&B2B_GTM=
https://www.wireless.miau.com/business/optin/optInEmailPopup.jsp?displayMode=1&from=businessCenter&bref=IBD903j370921100&wtLinkName=Subscribenow&wtLinkLoc=RN
https://www.wireless.miau.com/business/phones/phone_details_main.jsp?_DARGS=/business/templates/od_page_begin.jsp.pageLogoutForm
https://www.wireless.miau.com/business/policies/site_access.jsp;dsessionid=EBS51RW00NBS3B4R0ENCFEY?wtLinkName=TermsOfUse&wtLinkLoc=FTR
https://www.wireless.miau.com/business/shop/shop_cru_home.jsp?_DARGS=/business/shop/shop_cru_home.jsp.form_1
https://www.wireless.miau.com/pct/contract_acceptance/agreement.jsp?_DARGS=/pct/contract_acceptance/business_agreement_frag.jsp.pageBusinessForm
https://xdme.wireless.miau.com/jsp/login/login.jsp?Exp=Y&FP=Y
https://youachieve.miau.com/news.cfm?id=1204
I want to eliminate those that have several URLs with the same parameter but different values, I am trying this:
sort -u -t "=" -k1
and this:
sort -u -t "=" -k2
But I don't happen to succeed. Thank you very much

sort -u -t "=" -k1,1
will do what you want. The ,1 tells sort to stop at key position 1 too for the key. Default would be end-of-line, not end of field.

Related

How to get the number of row where I could find a specific value in RobotFramework?

I already written something like that:
Check privileges for PDTA
${end}= Get Matching Xpath Count //*[#id="listForm:displayDataTable:tbody"]/tr
${start}= Set Variable 0
: FOR ${index} IN RANGE ${start} ${end}
\ ${status}= Run Keyword And Ignore Error Element Should Contain listForm:displayDataTable:${index} su ${index}
And the log output is:
As you can see I want to get the number of row where I could find the value 'su'. This value can be found in a row number 6. Variable ${end} equals the number of all rows in the table.
Does anyone know how to get that number? Maybe there's a keyword which could help me, isn't it? Thanks in advance !!!
: FOR ${index} IN RANGE ${start} ${end}
\ ${Name}= Get Text listForm:displayDataTable:${index}
\ ${IsEqual}= Run Keyword And Return Status Should Be Equal ${Name} Su
\ ${RowNumber}= Set Variable ${index}
\ Run Keyword If '${IsEqual}'=='True' Run Keywords Log Rownumber is ${RowNumber} AND Exit For Loop
U can try this.
The variable ${RowNumber} gets the row number which has the text "Su".
As part of your FOR loop, I would add:
Run Keyword If '${status}' == 'PASS' Log ${index}
If you need to actually use it then simply set a variable or append to a list variable or something

VIM - Hotkey for VIMGREP on Visual Selection in current buffer

How would I go about setting up a hotkey (eg: CTRL+g) to perform a VIMGREP operation on the current visual selection in the current buffer? My intent is to show a line-numbered list in the "quickfix" window of all matching search results.
Right now, if I want to get a list of results for a regex search, I could do a command-mode query like so:
:vimgrep /foo/ %
However, there are two problems with this:
I don't want to have to type out the entire query. I could always do a visual selection, then use CTRL+r, CTRL+w, to paste the current visual selection into the command buffer, but I'd like something simpler than this.
The above approach requires that the current buffer is already saved to a file. I'd like to be able to work on a temporary buffer I've pasted into VIM rather than having to save a file buffer each time I want to do this.
Thank you.
A low-level solution
Try [I and the :ilist command:
[I " lists every occurrence of the word under the cursor
" in the current buffer (and includes)
:ilist /foo<CR> " lists every occurrence of foo in the current buffer
" (and includes)
Press : followed by a line number and <CR> to jump to that line.
You can use them on the visual selection with a simple mapping:
xnoremap <key> "vy:<C-u>ilist /<C-r>v<CR>:
You'll probably need to sanitize the register upon insertion, though.
See :help :ilist.
Another even lower-level solution
Since we are at it, let's dig even deeper and find the amazingly simple and elegant:
:g/foo/#
that you could use in the same way as :ilist above:
xnoremap <key> "vy:<C-u>g/<C-r>v/#<CR>:
Limitations
The solutions above don't use the quickfix window, obviously, but they allow you to:
see their result as a list,
use line numbers to actually get to where you want.
They have limitations, though:
the list is not cached so you must perform the search again if you want to get to a different occurrence,
the list is not transient like the quickfix list so you can't use navigation commands like :cnext or :clast to move around the result.
A higher-level solution
If those limitations are a showstopper, the function below, adapted from justinmk's answer in this /r/vim thread, gives you an almost complete solution:
press [I in normal mode to search for the word under the cursor in the whole buffer,
press ]I in normal mode to search for the word under the cursor after the current line,
press [I in visual mode to search for the selected text in the whole buffer,
press ]I in visual mode to search for the selected text after the current line.
The function below uses the quickfix list/window when the buffer is associated to a file and falls back to the regular behavior of [I and ]I otherwise. It could probably be modified to be used as part of an :Ilist command.
" Show ]I and [I results in the quickfix window.
" See :help include-search.
function! Ilist_qf(selection, start_at_cursor)
" there's a file associated with this buffer
if len(expand('%')) > 0
" we are working with visually selected text
if a:selection
" we build a clean search pattern from the visual selection
let old_reg = #v
normal! gv"vy
let search_pattern = substitute(escape(#v, '\/.*$^~[]'), '\\n', '\\n', 'g')
let #v = old_reg
" and we redirect the output of our command for later use
redir => output
silent! execute (a:start_at_cursor ? '+,$' : '') . 'ilist /' . search_pattern
redir END
" we are working with the word under the cursor
else
" we redirect the output of our command for later use
redir => output
silent! execute 'normal! ' . (a:start_at_cursor ? ']' : '[') . "I"
redir END
endif
let lines = split(output, '\n')
" better safe than sorry
if lines[0] =~ '^Error detected'
echomsg 'Could not find "' . (a:selection ? search_pattern : expand("<cword>")) . '".'
return
endif
" we retrieve the filename
let [filename, line_info] = [lines[0], lines[1:-1]]
" we turn the :ilist output into a quickfix dictionary
let qf_entries = map(line_info, "{
\ 'filename': filename,
\ 'lnum': split(v:val)[1],
\ 'text': getline(split(v:val)[1])
\ }")
call setqflist(qf_entries)
" and we finally open the quickfix window if there's something to show
cwindow
" there's no file associated with this buffer
else
" we are working with visually selected text
if a:selection
" we build a clean search pattern from the visual selection
let old_reg = #v
normal! gv"vy
let search_pattern = substitute(escape(#v, '\/.*$^~[]'), '\\n', '\\n', 'g')
let #v = old_reg
" and we try to perform the search
try
execute (a:start_at_cursor ? '+,$' : '') . 'ilist /' . search_pattern . '<CR>:'
catch
echomsg 'Could not find "' . search_pattern . '".'
return
endtry
" we are working with the word under the cursor
else
" we try to perform the search
try
execute 'normal! ' . (a:start_at_cursor ? ']' : '[') . "I"
catch
echomsg 'Could not find "' . expand("<cword>") . '".'
return
endtry
endif
endif
endfunction
nnoremap <silent> [I :call Ilist_qf(0, 0)<CR>
nnoremap <silent> ]I :call Ilist_qf(0, 1)<CR>
xnoremap <silent> [I :<C-u>call Ilist_qf(1, 0)<CR>
xnoremap <silent> ]I :<C-u>call Ilist_qf(1, 1)<CR>
NB: <C-r><C-w> inserts the word under the cursor, not the visual selection for which there's unfortunately no such shortcut. We have no choice but to yank.
Grepping a scratch buffer
You can use the :global command combined with :caddexpr to add entries to the current quickfix list. Here is the example from :h :caddexpr:
:g/mypattern/caddexpr expand("%") . ":" . line(".") . ":" . getline(".")
There are a few issues with this:
This approach only does one match per line
Does not start a new quickfix list
Really long to type out
Assumes the default global 'errorformat' hasn't been changed
To overcome these issues (all but the multiple matches per line) put the following command in your ~/.vimrc file:
command! -nargs=1 -bar Cgrep
\ let s:errorformat = &errorformat |
\ try |
\ let &errorformat='%f:%l:%m' |
\ cexpr [] |
\ execute 'g'.<q-args>.'caddexpr expand("%").":".line(".").":".getline(".")' |
\ cc |
\ finally |
\ let &errorformat = s:errorformat |
\ endtry
Now you can use :Cgrep/foo/ to grep the current buffer.
A visual mapping
To make it so you can do a visual version of this you need to yank in the selected text and pass it to our :Cgrep command via <c-r>. Here is an example visual mapping g/ to do just that:
xnoremap g/ y:<c-u>Cgrep/<c-r>"/<cr>
There are some issues with this mapping too:
This clobber the unnamed register, #"
This assumes the visually selected text will be a valid pattern
The following mapping fixes the mapping by using the expression register via <c-r>= and very no magic, \V:
xnoremap g/ :<c-u>let #/=#"<cr>gvy:let [#/,#"]=[#",#/]<cr>Cgrep/\V<cr>=substitute(escape(#/,'/\'),'\n','\\n','g')<cr>/<cr>
Conclusion
Personally I would forgo the mapping and get a visual star plugin (there are few out there). There is a nice Vimcast about this: Search for the selected text. I would then use that with the :Cgrep command we just created via :Cgrep// or better yet xmap g/ *:Cgrep//<cr>
For more help see:
:h :caddexpr
:h :cexpr
:h :g
:h 'efm
:h registers
:h /\V
To search for the visual selected text in files *.c via the hotkey CTRL+g do:
:vmap <silent> <unique> <c-g> y:vimgrep "<c-r>"" *.c<CR>
Two problems remain:
You want to search in a buffer, not in a file.
You want line numbers in the quickfix.
To 1:
As far as I know vimgrep can only search in files. The solution would be to write the buffer in a temporary file and search in this file and delete the temporary file when it is no longer needed. This solution requires a script which is called via the hotkey.
Just a hint: To get a suitable filename the VIM function tempname() can be used. Example:
let s:tmpfile = tempname()
(Sorry, I currently have no time to show a script solution here, maybe I add one later. Maybe somebody else has a better solution or can provide a script?)
To 2:
This command will enable line number in the current buffer:
:set number

Run a script on variables from a list file

I have a script which includes the following step as the first step in a series of filters of genomics data:
--option ~/folder$1/file$1 --option2 ~/folder$1/file$1 --indv NA12775 --options...
The script already uses a for-loop to go through folder/file indices 1-22. The option --indv takes a string, which is an identifiers. I have a separate list file which is just a column of "indv" identifiers:
NA06984
NA06986
NA06989
NA06994
NA07000
I have many such lists and I am looking for a solution to automatically take a single identifier from my list file, run the filtering script for "indv X" and then take the next consecutive identifier and repeat. Something like "for line in ID-list, run filter-script"...
You can use xargs for this:
xargs -I {} ./myprogram --indv {} < indvlist.txt
A couple bash methods for doing this:
for indv in $(<list-of-indv-values.txt)
do
something something .... ${indv} .....
done
or
while read indv
do
something something ... ${indv} .....
done < list-of-indv-values.txt

Use bash to concatenate a list of items

I have a list of items like:
ERR001268_chr6
ERR001312_chr6
ERR001332_chr6
ERR001361_chr6
ERR001369_chr6
ERR001413_chr6
ERR001433_chr6
ERR001462_chr6
ERR001698_chr6
ERR001734_chr6
ERR001763_chr6
ERR001774_chr6
ERR001799_chr6
say now I want to concatenate ERR001268_chr6 until ERR001763_chr6.
I can do cat ERR001268_chr6 ERR001269_chr6....ERR001763_chr6 > xxx
But obviously I don't want to type in these items one by one...So any simple bash commands to do this?
thx
Assuming that the item list is the full list of 'files' under current directory:
cat `ls -1 ERR*_chr6 | head -n11` > xxx

Makefile - loop through a list and select a specific value

I'm trying to do something like this (assuming $input is something provided by the user):
LIST = pre1 pre2 pre3 pre4 pre5 pre6 pre7 pre8 pre9 pre10
START = 0
for prefix in $(LIST); do \
if $(input) == $(prefix) then
START = 1
endif \
if $(START) == 1 then \
if [ -f $(prefix)<file_name> ]; then <do_A>; else <do_B>; fi
endif \
done
my problem is with the two if's mentioned above. i don't know how can i choose a specific string value from a list while iterating it (if $(input) == $(prefix) then) and i don't know how to check if a value is 1 or 0 (if $(START) == 1 then).
My intent with this code is to use the same makefile for different directories which have the same file name, but with a different prefix. sometimes, a directory might contain multiple versions of the file with a different prefix and i want to define a hierarchy of those prefixes (defined by LIST in my example). when the user provide a version, the idea is to start searching for the most up-to date version, starting from the version he provides (e.g. if the user provide pre4, then i need to search pre4 first and if it's not exist - i'll go on and search for pre5 and so on. but in this example, i won't search for pre1 even if it do exist in the current directory).
Anyone has an idea on how can i do that?
Thanks in advance.
If that is supposed to be a command in a Makefile, the syntax would have to be something like this:
LIST = pre1 pre2 pre3 pre4 pre5 pre6 pre7 pre8 pre9 pre10
START = 0
input = somename
file_name = whatever
some_target:
for prefix in $(LIST); do \
if test "$(input)" = $$prefix; then \
START=1; \
fi; \
if test "$(START)" = 1; then \
if test -f $$prefix$(file_name); then \
<do_A>; \
else \
<do_B>; \
fi; \
fi; \
done
But you didn't tell us what <input> and <file_name> are supposed to be, so I assumed they are other make variables. Basically the make rules look like one long shell line, with commands separated by semicolons, and lines continued with backslashes. $$ is replaced by make with a single $, which is why references to shell variables ($$prefix) need two dollars.
Your make manual (type man make has the whole story and is fun to read and understand.) Become a make guru today! Be sure to understand the difference between a make variable and a shell variable.