Could anyone please advise on a way to extract surnames that have spaces in them, as a single block of names?
I have names in a dataset that look like this
clear
input str40 name
"R. P. de la Espriella Guerrero"
"J. de Carvalho Ponce"
"E. De Freitas Drumond"
"R. de la Fuente and M. E. Medina-Mora"
"C. Van Heyningen and I. D. Watson"
"A. Z. van de Wiel and D. W. de Lange"
end
I only want the first surname (so only the first author and excluding other authors) but I want those names that have spaces to be extracted 'en bloc'. So, ultimately resulting in a dataset as follows, for instance:
clear
input str40 name
"de la Espriella Guerrero"
"de Carvalho Ponce"
"De Freitas Drumond"
"de la Fuente"
"Van Heyningen"
"van de Wiel"
end
I'd be grateful for any help.
Here is code that implements the two rules given in my comment above. It assumes the version of Stata used supports the unicode character string functions.
clear
input str40 name
"R. P. de la Espriella Guerrero"
"J. de Carvalho Ponce"
"E. De Freitas Drumond"
"R. de la Fuente and M. E. Medina-Mora"
"C. Van Heyningen and I. D. Watson"
"A. Z. van de Wiel and D. W. de Lange"
end
generate surname = name
replace surname = usubstr(surname,1,ustrpos(surname+" and "," and ")-1)
list, clean noobs
replace surname = usubstr(surname,ustrrpos(surname,". ")+1,.)
list, clean noobs
To learn Regex, I was solving some problems to train and study. And this is the problem, i know it might not be the best way to do with Regex, and my Regex is a mess, but i liked the challenge.
Problem:
The names needs to be Title Case;
There are exceptions for some lowercase words inside;
And some Names, e.g.: McDonald, MacDuff, D'Estoile
Names with ' and - are accepted, and sometimes they are o'Brien, O'brien, O'Brien, O' Brien or 'Ehu Kali.
No whitespaces on the beggining and end of Name;
No more than one space between each Name of Full Name;
A . is accepted if not alone, e.g.: Dan . Ferdnand (isn't accepted) and Dan G. Ferdnand (is accepted)
Numbers and symbols are not accepted
However, Roman numbers are accepted and aren't Title Case, e.g.: Elizabeth II
Some names can be alone, e.g.: Akihito (Prince of Japan)
Some special characters common in some countries are accepted, e.g.: Valeh ßlÿsgÿroğlu, Lażżru Role, Alaksiej Taraškievič
Regex
The code is
^(?![ ])(?!.*(?:\d|[ ]{2}|[!$%^&*()_+|~=`\{\}\[\]:";<>?,\/]))(?:(?:e|da|do|das|dos|de|d'|la|las|el|los|l'|al|of|the|el-|al-|di|van|der|op|den|ter|te|ten|ben|ibn)\s*?|(?:[A-ZàáâäãåąčćęèéêëėįìíîïłńòóôöõøùúûüųūÿýżźñçčšžÀÁÂÄÃÅĄĆČĖĘÈÉÊËÌÍÎÏĮŁŃÒÓÔÖÕØÙÚÛÜŲŪŸÝŻŹÑßÇŒÆČŠŽ∂ð'][^\s]*\s*?)(?!.*[ ]$))+$
And the Regex101 with a validation list
References
What i tried so far was based on these:
regular expression for first and last name
Regular Expression to disallow two consecutive white spaces in the middle of a string
A regex to test if all words are title-case
How to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loops
Use Regex to Split Numbered List array into Numbered List Multiline
Not working
I did this Regex and don't know how to make a way for it to not recognize the cases below, that are matching:
CAPITAL LETTER
AlTeRnAtE LeTtEr
And those aren't and should:
Urxan Əbűlhəsənzadə
İsmət Jafarov
Şükür Hagverdiyev
Űmid Abdurrahimov
Ġerardo Seralta
Ċikku Paris
Question
Is there a way to optimize this Regex (monster)?
And how do i fix the problems stated before on Not working?
p.s.: The list of names with examples for validation can be found on the link to Regex101.
Brief
Seeing as how you're learning Regex and haven't specified a regex flavour to use, I've chosen PCRE as it has a wide variety of support in the regex world.
Code
See this regex in use here
(?(DEFINE)
(?# Definitions )
(?<valid_nameChars>[\p{L}\p{Nl}])
(?<valid_nonNameChars>[^\p{L}\p{Nl}\p{Zs}])
(?<valid_startFirstName>(?![a-z])[\p{L}'])
(?<valid_upperChar>(?![a-z])\p{L})
(?<valid_nameSeparatorsSoft>[\p{Pd}'])
(?<valid_nameSeparatorsHard>\p{Zs})
(?<valid_nameSeparators>(?&valid_nameSeparatorsSoft)|(?&valid_nameSeparatorsHard))
(?# Invalid combinations )
(?<invalid_startChar>^[\p{Zs}a-z])
(?<invalid_endChar>.*[^\p{L}\p{Nl}.\p{C}]$)
(?<invalid_unaccompaniedSymbol>.*(?&valid_nameSeparatorsHard)(?&valid_nonNameChars)(?&valid_nameSeparatorsHard))
(?<invalid_overTwoUpper>(?:(?&valid_nameChars)*\p{Lu}){3})
(?<invalid>(?&invalid_startChar)|(?&invalid_endChar)|(?&invalid_unaccompaniedSymbol)|(?&invalid_overTwoUpper))
(?# Valid combinations )
(?<valid_name>(?:(?:(?&valid_nameChars)|(?&valid_nameSeparatorsSoft))*(?&valid_nameChars)+(?:(?&valid_nameChars)|(?&valid_nameSeparatorsSoft))*)+\.?)
(?<valid_firstName>(?&valid_startFirstName)(?:\.|(?&valid_name)*))
(?<valid_multipleName>(?&valid_firstName)(?=.*(?&valid_nameSeparators)(?&valid_upperChar))(?:(?&valid_nameSeparatorsHard)(?&valid_name))+)
(?<valid>(?&valid_multipleName)|(?&valid_firstName))
)
^(?!(?&invalid))(?&valid)$
Results
Input
== 1NcOrrect N4M3S ==
CAPITAL LETTER
AlTeRnAtE LeTtEr
Natalia maria
Natalia aria
Natalia orea
Maria dornelas
Samuel eto'
Miguel lasagna
Antony1 de Home Ap*ril
Ap*ril Willians
Antony_ de Home Apr+il
Ant_ony de Home Apr#il
Antony# de Ho#me Apr^il
Maria Silva
Maria silva
maria Silva
Maria Silva
Maria Silva
Maria / Silva
Maria . Silva
John W8
==Correct Names==
Urxan Əbűlhəsənzadə
İsmət Jafarov
Şükür Hagverdiyev
Űmid Abdurrahimov
Ġerardo Seralta
Ċikku Paris
Hind ibn Sheik
Colop-U-Uichikin
Lażżru Role
Alaksiej Taraškievič
Petruso Husoǔski
Sumu-la-El
Valeh ßlÿsgÿroğlu
'Arab al-Rashayida
Tariq al-Hashimi
Nabeeh el-Mady
Tariq Al-Hashimi
Brian O'Conner
Maria da Silva
Maria Silva
Maria G. Silva
Maria McDuffy
Getúlio Dornelles Vargas
Maria das Flores
John Smith
John D'Largy
John Doe-Smith
John Doe Smith
Hector Sausage-Hausen
Mathias d'Arras
Martin Luther King Jr.
Ai Wong
Chao Chang
Alzbeta Bara
Marcos Assunção
Maria da Silva e Silva
Juscelino Kubitschek de Oliveira
Maria da Costa e Silva
Samuel Eto'o
María Antonieta de las Nieves
Eugène
Antòny de Homé April
àntony de Home ùpril
Antony de Home Aprìl
Pierre de l'Estache
Pierre de L'Estoile
Akihito
Nadine Schröder
Anna A. Møller
D. Pedro I
Pope Benedict XVI
Marsibil Ragnarsdóttir
Natanaël Morel
Isaac De la Croix
Jean-Michel Bozonnet
Qutaibah Mu'tazz Abadi
Rushd Jawna' Kassab
Khaldun Abdul-Qahhar Sabbag
'Awad Bashshar Asker
Al B. Zellweger
Gunnleif Snæ-Ulfsson
Käre Toresson
Sorli Ærnmundsson
Arnkel Øystæinsson
Ástríður Dórey
Åsmund Kåresson
Yahatti-Il
Ipqu-Annunitum
Nabu-zar-adan
Eskopas Cañaverri
Botolph of Langchester
Aelfhun the Cantrell
Fraco di Natale
Fraco Di Natale
Iván de Luca
Iván De Luca
Man'nah
Atabala Aüamusalü
Ramiz Ağasəfalu
Dadaş Aghakhanov
Fÿrxad Mübarizlı
Vaclaǔ Šupa
Yakiv Volacič
Flor Van Vaerenbergh
Flor van Vaerenbergh
Edwin van der Sar
Husein Ekmečić
Álvaro Guimarães Alencar
Phone U Yaza Arkar
Seocan MacGhille
X'wat'e Tlekadugovy
Albert-Jan Bootsveld
Maurits-jan Kuipers op den Kollenstaart
Elco ter Hoek
Robbert te Poele
Aad ten Have
'Ehu Kali
Ho'opa'a Loni
Aukanai'i Mahi'ai
Kalman ben Tal El
Żytomir Roszkowski
K'awai
==EXTRA== only if possible, strange ones
Maol-Moire Mac'IlleBhuidh
Tòmas MacIlleChruim
Aindreas MacIllEathain
Eanruig MacGilleBhreac
Peadar MacGilleDhonaghart
Maolmhuire MacGill-Eain
Eanruig MacGilleBhreac
Wim van 't Plasman
Output
Note: Shown below are only the strings that matched from the above Input
Urxan Əbűlhəsənzadə
İsmət Jafarov
Şükür Hagverdiyev
Űmid Abdurrahimov
Ġerardo Seralta
Ċikku Paris
Hind ibn Sheik
Colop-U-Uichikin
Lażżru Role
Alaksiej Taraškievič
Petruso Husoǔski
Sumu-la-El
Valeh ßlÿsgÿroğlu
'Arab al-Rashayida
Tariq al-Hashimi
Nabeeh el-Mady
Tariq Al-Hashimi
Brian O'Conner
Maria da Silva
Maria Silva
Maria G. Silva
Maria McDuffy
Getúlio Dornelles Vargas
Maria das Flores
John Smith
John D'Largy
John Doe-Smith
John Doe Smith
Hector Sausage-Hausen
Mathias d'Arras
Martin Luther King Jr.
Ai Wong
Chao Chang
Alzbeta Bara
Marcos Assunção
Maria da Silva e Silva
Juscelino Kubitschek de Oliveira
Maria da Costa e Silva
Samuel Eto'o
María Antonieta de las Nieves
Eugène
Antòny de Homé April
àntony de Home ùpril
Antony de Home Aprìl
Pierre de l'Estache
Pierre de L'Estoile
Akihito
Nadine Schröder
Anna A. Møller
D. Pedro I
Pope Benedict XVI
Marsibil Ragnarsdóttir
Natanaël Morel
Isaac De la Croix
Jean-Michel Bozonnet
Qutaibah Mu'tazz Abadi
Rushd Jawna' Kassab
Khaldun Abdul-Qahhar Sabbag
'Awad Bashshar Asker
Al B. Zellweger
Gunnleif Snæ-Ulfsson
Käre Toresson
Sorli Ærnmundsson
Arnkel Øystæinsson
Ástríður Dórey
Åsmund Kåresson
Yahatti-Il
Ipqu-Annunitum
Nabu-zar-adan
Eskopas Cañaverri
Botolph of Langchester
Aelfhun the Cantrell
Fraco di Natale
Fraco Di Natale
Iván de Luca
Iván De Luca
Man'nah
Atabala Aüamusalü
Ramiz Ağasəfalu
Dadaş Aghakhanov
Fÿrxad Mübarizlı
Vaclaǔ Šupa
Yakiv Volacič
Flor Van Vaerenbergh
Flor van Vaerenbergh
Edwin van der Sar
Husein Ekmečić
Álvaro Guimarães Alencar
Phone U Yaza Arkar
Seocan MacGhille
X'wat'e Tlekadugovy
Albert-Jan Bootsveld
Maurits-jan Kuipers op den Kollenstaart
Elco ter Hoek
Robbert te Poele
Aad ten Have
'Ehu Kali
Ho'opa'a Loni
Aukanai'i Mahi'ai
Kalman ben Tal El
Żytomir Roszkowski
K'awai
Maol-Moire Mac'IlleBhuidh
Tòmas MacIlleChruim
Aindreas MacIllEathain
Eanruig MacGilleBhreac
Peadar MacGilleDhonaghart
Maolmhuire MacGill-Eain
Eanruig MacGilleBhreac
Wim van 't Plasman
Explanation
I used a define block to create definitions. You can look at each definition to see how it works. In general, I use \p{.} where . is replaced with some pointer to a Unicode character group (i.e \p{L} is any letter from any language - this will not work in most flavours of regex, but it does allow the regex to be much more simplified if available, which is why I used it).
If you need anything else explained, don't hesitate to ask me and I'll do my best, but regex101 should be able to explain anything you're wondering about regex.
I would like to get the text in the frame "[[Fichier .... ]]" here in the text :
=== Langues ===
{{Article détaillé|Langues en Afrique du Sud}}
[[Fichier:South Africa dominant language map.svg|thumb|300px| Répartition
des langues officielles dominantes par région :
{{clear}}
{{legend|#80b1d3|[[Zoulou]]}}
{{legend|#8dd3c7|[[Afrikaans]]}}
{{legend|#fb8072|[[Xhosa (langue)|Xhosa]]}}
{{legend|#ffffb3|[[Anglais]]}}
{{legend|#fccde5|[[Tswana|Setswana]]}}
{{legend|#bebada|[[Ndébélés|Ndebele]]}}
{{legend|#fdb462|[[Sotho du Nord]]}}
{{legend|#b3de69|[[Sotho du Sud]]}}
{{legend|#bc80bd|[[Swati]]}}
{{legend|#ccebc5|[[Venda (langue)|Tshivenda]]}}
{{legend|#ffed6f|[[Tsonga (langue)|Xitsonga]]}}
{{legend|#d0d0d0|Pas de langage dominant}}]]
Il n'y a pas de langue maternelle majoritairement dominante en Afrique du Sud. Depuis [[1994]], [[Langues en Afrique du Sud|onze langues officielles]] (anglais, afrikaans, zoulou, xhosa, zwazi, ndebele, sesotho, sepedi, setswana, xitsonga, tshivenda<ref>[http://www.lafriquedusud.com/ethnies.htm lafriquedusud.com]</ref>) sont reconnues par la [[Constitution de l'Afrique du Sud|Constitution sud-africaine]]<ref>{{Ouvrage|langue=fr|auteur1=François- Xavier Fauvelle-Aymar|titre=Histoire
How can I improve the following regex:
\[\[Fichier:.*(.*\[\[.*\]\].*)*.*\]\]
In order to match all the liness until the correct ]]?
\[\[Fichier:(.*?(\n))+.*\]\]
Match all the lines between [[ and ]].
Here is the best sandbox: http://www.regexr.com
Provided you my have at most one level of nested [[...]] (as your test data sample suggests), the inner regex pattern may comprise a sequence of either a string in double brackets (\[\[.*?\]\]) or anything but a closing bracket ([^]]):
\[\[Fichier:(?:\[\[.*?\]\]|[^]])*\]\]
Demo: https://regex101.com/r/Q7zQQt/1
For arbitrary number of nested levels the answer depends on regex flavour. You may find more details on this here: http://www.regular-expressions.info/balancing.html.
I have a ".txt" file which came with a lot of juridics text but I only want to extract the dates to make a further analysis and graphics. Here is an example (sorry its in Portuguese):
"AR - 4024-03.2010.5.00.0000(2)" "ACORDAM os Ministros da Egrégia
Subseção II Especializada em Dissídios Individuais do Tribunal
Superior do Trabalho, por unanimidade, não conhecer do recurso
ordinário, por incabível. Brasília, 24 de maio de 2011. Firmado por
assinatura digital (MP 2.200-2/2001) Alberto Luiz Bresciani de Fontan
Pereira Ministro Relator fls. PROCESSO Nº
TST-AR-4024-03.2010.5.00.0000 Firmado por assinatura digital em
26/05/2011 pelo sistema AssineJus da Justiça do Trabalho, conforme MP
2.200-2/2001, que instituiu a Infra-Estrutura de Chaves Públicas Brasileira."
That file has a lot of those things but I want to extract only the highlighted parts and put them in a separate vector. I've been trying match, grep nothing is working. Perhaps because I'm new to R.
This pattern will match dates of the form that you have highlighted:
"\\d{1,2} de (janeiro|fevereiro|março|abril|maio|junho|julho|agosto|septembro|outubro|novembro|dezembro) de \\d{4}"
See ?regex for details on special characters and quantifiers. You can substitute on the items that match:
your_text <- c("AR - 4024-03.2010.5.00.0000", "ACORDAM os Ministros da Egrégia Subseção II Especializada em Dissídios Individuais do Tribunal Superior do Trabalho, por unanimidade, não conhecer do recurso ordinário, por incabível. Brasília, 24 de maio de 2011. Firmado por assinatura digital (MP 2.200-2/2001) Alberto Luiz Bresciani de Fontan Pereira Ministro Relator fls. PROCESSO Nº TST-AR-4024-03.2010.5.00.0000 Firmado por assinatura digital em 26/05/2011 pelo sistema AssineJus da Justiça do Trabalho, conforme MP 2.200-2/2001, que instituiu a Infra-Estrutura de Chaves Públicas Brasileira.")
sub( "(.+ )(\\d{1,2} de (janeiro|fevereiro|março|abril|maio|junho|julho|agosto|septembro|outubro|novembro|dezembro) de \\d{4})(.+)", "\\2", your_text[grepl("\\d{1,2} de (janeiro|fevereiro|março|abril|maio|junho|julho|agosto|septembro|outubro|novembro|dezembro) de \\d{4}", your_text )
[1] "AR - 4024-03.2010.5.00.0000" "24 de maio de 2011"
To remove the non-date-containing items, you can use grepl to preselect:
> sub( "(.+ )(\\d{1,2} de (janeiro|fevereiro|março|abril|maio|junho|julho|agosto|septembro|outubro|novembro|dezembro) de \\d{4})(.+)", "\\2", your_text[grepl("\\d{1,2} de (janeiro|fevereiro|março|abril|maio|junho|julho|agosto|septembro|outubro|novembro|dezembro) de \\d{4}", your_text )])
[1] "24 de maio de 2011"
If you need to play with the patterns to get the hang of using capture-classes, there are nifty regex testing webpages.
I have a CSV file like that:
"","LESCHELLES","","LESCHELLES"
"","SAINTE CROIX DE VERDON","","SAINTE CROIX DE VERDON"
"","SERRE CHEVALIER","","SERRE CHEVALIER"
"","SAINT JUST D'ARDECHE","","SAINT JUST D'ARDECHE"
"","NEUVILLE SUR VANNES","","NEUVILLE SUR VANNES"
"","ESCUEILLENS ET SAINT JUST","","ESCUEILLENS ET SAINT JUST"
"","PAS DES LANCIERS","","PAS DES LANCIERS"
"","PLAN DE CAMPAGNE","","PLAN DE CAMPAGNE"
And I'd like to convert it this way:
"","Leschelles","","LESCHELLES"
"","Sainte Croix De Verdon","","SAINTE CROIX DE VERDON","STE CROIX DE VERDON","93"
"","Serre Chevalier","","SERRE CHEVALIER","SERRE CHEVALIER","93"
"","Saint Just D'Ardeche","","SAINT JUST D'ARDECHE"
"","Neuville Sur Vannes","","NEUVILLE SUR VANNES"
"","Escueillens Et Saint Just","","ESCUEILLENS ET SAINT JUST","ESCUEILLENS ET ST JUST","91"
"","Luc","","LUC"
"","Pas Des Lanciers","","PAS DES LANCIERS","PAS DES LANCIERS","93"
"","Plan De Campagne","","PLAN DE CAMPAGNE","PLAN DE CAMPAGNE","93"
This would be nice. And better: lowercase all "whole" words like de, d', et, sur and des. This would give:
"","Leschelles","","LESCHELLES"
"","Sainte Croix de Verdon","","SAINTE CROIX DE VERDON","STE CROIX DE VERDON","93"
"","Serre Chevalier","","SERRE CHEVALIER","SERRE CHEVALIER","93"
"","Saint Just d'Ardeche","","SAINT JUST D'ARDECHE"
"","Neuville sur Vannes","","NEUVILLE SUR VANNES"
"","Escueillens et Saint Just","","ESCUEILLENS ET SAINT JUST","ESCUEILLENS ET ST JUST","91"
"","Luc","","LUC"
"","Pas des Lanciers","","PAS DES LANCIERS","PAS DES LANCIERS","93"
"","Plan de Campagne","","PLAN DE CAMPAGNE","PLAN DE CAMPAGNE","93"
Python has title():
Return a titlecased version of the string where words start with an
uppercase character and the remaining characters are lowercase.
The algorithm uses a simple language-independent definition of a word
as groups of consecutive letters. The definition works in many
contexts but it means that apostrophes in contractions and possessives
form word boundaries, which may not be the desired result:
"they're bill's friends from the UK".title() "They'Re Bill'S Friends From The Uk"
A workaround for apostrophes can be constructed
using regular expressions:
import re
def titlecase(s):
return re.sub(r"[A-Za-z]+('[A-Za-z]+)?",
lambda mo: mo.group(0)[0].upper() +
mo.group(0)[1:].lower(),
s)
titlecase("they're bill's friends.") "They're Bill's Friends."
Update: here's the solution for French problem:
import re, sys
def titlecase(s):
return re.sub(r"[A-Za-z]+('[A-Za-z]+)?",
lambda mo: mo.group(0)[0].upper() +
mo.group(0)[1:].lower(),
s)
def french_parse(s):
p = re.compile(
r"( de la | sur | sous | la | de | les | du | le | au | aux | en | des | et )|(( d'| l')([a-z]+))",
re.IGNORECASE)
return p.sub(
lambda mo: mo.group().find("'")>0
and mo.group()[:mo.group().find("'")+1].lower() +
titlecase(mo.group()[mo.group().find("'")+1:])
or (mo.group(0)[0].upper() + mo.group(0)[1:].lower()),
s);
for line in sys.stdin:
s = line[20:len(line)-1]
p = s.find('"')
t = s[:p]
# Just output to show which names have been modified:
if french_parse( titlecase(t) ) != titlecase(t):
print '"' + french_parse( titlecase(t) ) + '"'
Just launch it like this:
python thepythonscript.py < file.csv
Then the output will be:
"Grenand les Sombernon"
"Touville sur Montfort"
"Fontenay en Vexin"
"Durfort Saint Martin de Sossenac"
"Monclar d'Armagnac"
"Ports sur Vienne"
"Saint Barthelemy de Beaurepaire"
"Saint Bernard du Touvet"
"Rosoy le Vieil"
While you may be able to pull this off with some vim regex magic, I think it'll be easier if you solve the problem in your favorite scripting language, and pipe selected text through that from vim using the ! command. Here's an (untested) example in PHP:
#!/usr/bin/env php
<?php
$specialWords = array('de', 'd\'', 'et', 'du', /* etc. */ );
foreach (file('php://stdin') as $ville) {
$line = ucwords($line);
foreach ($specialWords as $w) {
$line = preg_replace("/\\b$w\\b/i", $w, $line);
}
echo $line;
}
Make that script executable and store it somewhere on your PATH; then from vim, select some text and use :'<,'>! yourscript.php to convert (or just :%! yourscript.php for the whole buffer).
The csv.vim ftplugin helps with working in CSV files. Though it does not offer a "substitute in column N" function directly, it may get your near that. At least you can arrange the columns into neat blocks, and then apply a simple regexp or visual blockwise selection to it.
But I second that using a different toolchain that is more suited to manipulating CSV-files may be preferable over doing this completely in Vim. It also depends on whether it's a one-off task or, you do this frequently.
Here is an oneliner vim command.
%s/"[^"]*",\zs\("[^"]*"\)/\=substitute(substitute(submatch(0), '\<\(\a\)\(\a*\)\>', '\u\1\L\2', 'g'), '\c\<\(de\|d\|l\|sur\|le\|la\|en\|et\)\>', '\L&', 'g')
I expect here to have no double-quotes in the first two fields.
The idea behind this solution is to rely on :h :s\= to execute a series of functions on the second field once found. The series of functions being: first change each word to TitleCase, then put all liants in lowercase.