Removing duplicate rows from Notepad++ - regex
I am looking for a way to remove duplicate rows from my Notepad++ file. The rows are not exact duplicates per say. Here's the situation. I have a large file of capitalized company names with probability values as well (each separated by a tab). So the format would be like this:
ATT .7213
SAMSUNG .01294
SAMSUNG .90222
So, I need to remove one of these rows because there is a match in the first column. I don't really have a preference of which one I need to remove just as long as I end up with one row at the end. I have tried to use unique sorting with TextFX but it's looking for the whole row duplicate and not just the first column. If anyone could offer up a handy solution to fix this I would greatly appreciate it. Bash script answers using awk, sed, or cut are also acceptable as well as using regular expressions.
Thank you!
Using awk, you could say:
awk '!a[$1]++' filename
This would keep only the lines having a unique value for the first field.
Use sort:
sort -k1,1 -u companies.txt
The output will consist of the full line, but only the sorting key (the first field) will be considered for identifying duplicates.
Related
How to Keep rows of multi-line cells containing a keyword in google sheets
I'm trying to keep lines that contain the word "NOA" in a column A which has many multi-line cells as can be viewed in this Google Spreadsheet. If "NOA" is present then, I would like to keep the line. The input and output should look like the image which I have "working" with too-many helper cells. Can this be combined into a single formula? Theoretical Approaches: I have been thinking about three approaches to solve this: ARRAYFORMULA(REGEXREPLACE - couldn't get it to work JOIN(FILTER(REGEXMATCH(TRANSPOSE - showing promise as it works in multiple steps Using the QUERY Function - unfamiliar w/ function but wondering if this function has a fast solution Practical attempts: FIRST APPROACH: first I attempted using REGEXEXTRACT to extract out everything that did not have NOA in it, the Regex worked in demo but didn't work properly in sheets. I thought this might be a concise way to get the value, perhaps if my REGEX skill was better? ARRAYFORMULA(REGEXREPLACE(A1:A7, "^(?:[^N\n]|N(?:[^O\n]|O(?:[^A\n]|$)|$)|$)+","")) I think the Regex because overly complex, didn't work in Google or perhaps the formula could be improved, but because Google RE2 has limitations it makes it harder to do certain things. SECOND APPROACH: Then I came up with an alternate approach which seems to work 2 stages (with multiple helper cells) but I would like to do this with one equation. =TRANSPOSE(split(A2,CHAR(10))) =TEXTJOIN(CHAR(10),1,FILTER(C2:C7,REGEXMATCH(C2:C7,"NOA"))) Questions: Can these formulas be combined and applied to the entire Column using an Index or Array? Or perhaps, the REGEX in my first approach can be modified? Is there a faster solution using Query? The shared Google spreadhseet is here. Thank you in advance for your help.
Here's one way you can do that: =index(substitute(substitute(transpose(trim( query(substitute(transpose(if(regexmatch(split( filter(A2:A,A2:A<>""),char(10)),"NOA"),split( filter(A2:A,A2:A<>""),char(10)),))," ","❄️") ,,9^9)))," ",char(10)),"❄️"," ")) First, we split the data by the newline (char 10), then we filter out the lines that don't contain NOA and finally we use a "query smush" to join everything back together.
How can I extract specific patterns from a string?
I currently have a dataset filled with the following pattern: My goal is to get each value into a different cell. I have tried with the following formula, but it's not yielded the results I am looking for. =SPLIT(D8,"[Stock]",FALSE,FALSE) I would appreciate any guidance on how I can get to the ideal output, using Google Sheets. Thank you in advance!
I will assume here from your post that your original data runs D8:D. If you want to retain [Stock] in each entry, try the following in the Row-8 cell of a column that is otherwise empty from Row 8 downward: =ArrayFormula(IF(D8:D="",,TRIM(SPLIT(REGEXREPLACE(D8:D&"~","(\[Stock\]).","$1~"),"~",1,1)))) If you don't want to retain [Stock] in each entry, use this version: =ArrayFormula(IF(D8:D="",,TRIM(SPLIT(REGEXREPLACE(D8:D&"~","\[Stock\].","~"),"~",1,1)))) These formulas don't function based on using any punctuation at all as markers. They also assure that you don't wind up with blank (and therefore unusable) cells interspersed for ending SPLITs.
, only used in the separator =ARRAYFORMULA(SPLIT(D8:D,", ",FALSE)) , used also in each string ([stock] will be replaced) =ARRAYFORMULA(SPLIT(D8:D," [Stock], ",FALSE)) , used also in each string ([stock] will not be replaced) =ArrayFormula(SPLIT(REGEXREPLACE(M9:M11,"(\[Stock\]), ","$1♦"),"♦"))
use: =INDEX(TRIM(IFNA(SPLIT(D8:D; ","))))
Excel- Extract Number from Cell
I have multiple cells that I am attempting to extract a number from, and need help finding a regex alternative.
The cells range in the following formats:
asdfs. Seat#29 asfddsa
asdfsa. Seat#5d
asdfasN/A . Seat#22 as789fsd
Seat#111 words33
The closest that I came to a solution is:
=IFERROR(TRIM(MID([#DisplayName],FIND("#",[#DisplayName])+1,3)),"")
As you can see this will extract most of the numbers but for some it leaves a character at the end.
The only commonality is the # preceding the seat number. I am trying to extract only the seat number, no other numbers.
I cannot use VBA, this must be done using formulas. I have figured this out once before but stupidly pasted over the formulas with a values only paste.
This can be done utilizing a flash fill, but I was hoping for a more stable formula.
If you want just the numbers then use:
=--MID(A1,FIND("#",A1)+1,AGGREGATE(15,6,ROW(1:5)/(ISERROR(--MID(REPLACE(A1,1,FIND("#",A1),""),ROW(1:5),1))),1)-1)
If you want the letter also then:
=MID(A1,FIND("#",A1)+1,FIND(" ",REPLACE(A1,1,FIND("#",A1),""))-1)
If you do not need the letter following the seat number, you can use .*#(\d+) Edit for clarity: Excel does not have regex functions built in. You will either have to use a UDF (I can help with that if you'd like) or use a non-regex solution.
Here is a solution without VBA to extract all numbers inside the strings. https://drive.google.com/open?id=1Fk6VFznD3i8s6scADy_vXCEj-1zQpBPW Sheet #3
Possible combination (variations) of words in a string variable in stata
I have a string variable containing school names and I need to find all the possible combination of each word in this string variable in stata: For example variation of a word "Academy" would be: Academy, Academy, acdamey, aacdemy, dmcaamy, aacedmy, and so on. I need this to standardize the raw data of school names, which has many typos of each word due to data entry issues, like the ones given above for "academy".
Depending whether your data is already in the Excel sheets or a file, you can either use regex trying to match all possible combinations (and probably fix them when found) or parse the strings first before bringing them into Excel. In either case you could make a file (or Excel list/table/area/etc.) that includes all the common typos and pick each typo as regex match to use when comparing to your actual input. Making regexp that would actually find all possible cases is next to impossible, especially if there are cases where very similar (but correct) names for schools exist. In any case direct regexps would be very messy and complex, so I would advice you to parse the data by finding first the correct form, excluding it and then using (greedy) search/regex to find the typoed versions. You can then save the typos to use them as a filter/match/pattern. To get some sort of starting ideas, check this links: Regex: Search for verb roots Read text file and extract string into Excel sheet using regex P.s You should keep the count of all strings/school names and finally get a list of all names that did not match correct form or any of your regexp filters, so you can manually insert/correct them.
Search/replace in block selection in Notepad++
Is there a way to limit search/replace only to a columnar block selection in Notepad++? Here is what I am trying to do: I am bulk-editing metadata extracted from large numbers of photos. The metadata comes to me as a csv file with no quotes around fields in header line and no quotes around first field in each succeeding line. I edit this file in Open Office calc which exports with quotes around all fields. I can easily edit header row but the problem comes in stripping quotes from only first field in successive lines. I can use notepad in columnar mode but, after selecting the first column, the 'search only in selection' option box is greyed out. I can do this by hand but it means lots of hand-work and increased chance of error.
I know, this probably won't help you any more, but I just had the same problem and stumbled across this question. I found moving the block in question to a new file and performing the find/replace there works quite decently. When moving the block back, be sure to select it in block mode (see this question).
No. Another editor may have this feature.
sort of a late reply but... I had the same problem when I moved to a new machine with Notepad++ installed. Previously, I was using a text editor called Boxer that had this feature, which I found invaluable. Its not free-ware however.
You may not be able to Search/Replace within a columnar selection, but you can easily carry out your task within Notepad++. Use Find and Replace feature, with the Regular Expressions box checked.
If you want to remove quotes only from a target column, use the following regular expression in the Find field:
(^([^,]*,){i})"([^,\n\r]*)"(.*$)
Replace i with the position of the target column minus 1.
(i.e.- Us 2 if you want quotes around the third column, 0 for the first column, etc)
In the Replace field use:
\1\3\4
Clicking "Replace All" will strip quotes from the target column.
If you want to blow away all quotes surrounding each element in your csv without prejudice, use the following regular expression in the Find field:
((?<=,)|(?<=^))"(.*?)"((?=$|,))
In the Replace field use:
\1\2\3
Clicking Replace All will strip quotes form the columns.
Example
Since you didn't provide an example csv file, I'll walk through my own working example. Below is my csv:
"0","1","2","3","4","5","6","7","8","9"
"10","11","12","13","14","15","16","17","18","19"
"20","21","22","23","24","25","26","27","28","29"
"30","31","32","33","34","35","36","37","38","39"
"40","41","42","43","44","45","46","47","48","49"
"50","51","52","53","54","55","56","57","58","59"
"60","61","62","63","64","65","66","67","68","69"
"70","71","72","73","74","75","76","77","78","79"
"80","81","82","83","84","85","86","87","88","89"
"90","91","92","93","94","95","96","97","98","99"
"100","101","102","103","104","105","106","107","108","109"
"110","111","112","113","114","115","116","117","118","119"
"120","121","122","123","124","125","126","127","128","129"
"130","131","132","133","134","135","136","137","138","139"
"140","141","142","143","144","145","146","147","148","149"
"150","151","152","153","154","155","156","157","158","159"
"160","161","162","163","164","165","166","167","168","169"
"170","171","172","173","174","175","176","177","178","179"
"180","181","182","183","184","185","186","187","188","189"
"190","191","192","193","194","195","196","197","198","199"
If I wanted to remove quotes from the second column, I would use the below Find and Replace fields
(^([^,]*,){1})"([^,\n\r]*)"(.*$)
\1"\3"\4
Clicking Replace All yields the below result:
"0",1,"2","3","4","5","6","7","8","9"
"10",11,"12","13","14","15","16","17","18","19"
"20",21,"22","23","24","25","26","27","28","29"
"30",31,"32","33","34","35","36","37","38","39"
"40",41,"42","43","44","45","46","47","48","49"
"50",51,"52","53","54","55","56","57","58","59"
"60",61,"62","63","64","65","66","67","68","69"
"70",71,"72","73","74","75","76","77","78","79"
"80",81,"82","83","84","85","86","87","88","89"
"90",91,"92","93","94","95","96","97","98","99"
"100",101,"102","103","104","105","106","107","108","109"
"110",111,"112","113","114","115","116","117","118","119"
"120",121,"122","123","124","125","126","127","128","129"
"130",131,"132","133","134","135","136","137","138","139"
"140",141,"142","143","144","145","146","147","148","149"
"150",151,"152","153","154","155","156","157","158","159"
"160",161,"162","163","164","165","166","167","168","169"
"170",171,"172","173","174","175","176","177","178","179"
"180",181,"182","183","184","185","186","187","188","189"
"190",191,"192","193","194","195","196","197","198","199"
My search on internet, to to see weather notepad++ suports this; brought me here. I have used TextPad and confirm that it supports find-and-replace within column selected block. Also TextPad is free for personal use.