split a string by commas, but commas in a token - regex

Let's say I have this:
something,"another thing"
This can be split easily with a normal split function.
Now I want to have more complicated syntax and I do:
something,"in a string, oooh",rgba(4,2,0)
This does not work with a regular split function.
I tried using things like replacing commas inside of specific types of tokens, but that became too over-complicated and I feel there has to be a better way.
Then I tried with regular expressions, which worked, until I had to add a new feature, which wouldn't work with the regexp I had (which was pretty bad), also regexp matches can be slow, and this is supposed to be as fast as possible.
What would be a better way to solve this?
Here is the source repo for extra context https://github.com/hyprland-community/hyprparse
And the format in question is the hyprland config format

Iterate over the string keeping a context state:
None
Inside a "..."
Inside a (...)
Inside a context, comma has no separator meaning.
Limitations: This is a midnight hack!
See also Rust Playground
fn split(s: String) -> Vec<String> {
let mut context = None;
let mut i = 0;
let mut start = 0;
let mut items = Vec::new();
for c in s.chars() {
if context == Some('"') {
if c == '"' {
context = None;
}
i = i+1;
continue;
} else if context == Some('(') {
if c == ')' {
context = None;
}
i = i+1;
continue;
}
if c == '"' || c == '(' {
context = Some(c);
}
if c == ',' && context.is_none() {
items.push(s[start..i].to_string());
start = i + 1;
}
i = i+1;
}
items.push(s[start..i].to_string());
items
}
fn main() {
let s = "something,\"in a string, oooh\",rgba(4,2,0)".to_string();
println!("{:?}", split(s));
// -> ["something", "\"in a string, oooh\"", "rgba(4,2,0)"]
}

Thanks for all the help everyone, I eventually came up with my own solution with the help of people I knew IRL, here is my solution:
fn previous_rgb(index: usize, chars: Vec<char>) -> bool {
let string: String = chars[index-3..index].iter().collect();
let string = string.as_str();
if string == "gba" || string == "rgb" {
true
} else {
false
}
}
fn splitter<Str: ToString>(s: Str) -> Vec<String> {
let mut is_in_rgb = false;
let mut is_in_str = false;
let mut last = 0;
let chars: Vec<_> = s.to_string().chars().collect();
let mut final_str = vec![];
for (index, c) in chars.iter().enumerate() {
if *c == '(' && previous_rgb(index, chars.clone()) && !is_in_str {
is_in_rgb = true;
} else if *c == ')' && is_in_rgb && !is_in_str {
is_in_rgb = false;
} else if *c == '"' && is_in_str {
is_in_str = false;
} else if *c == '"' && !is_in_str {
is_in_str = true;
} else if *c == ',' && !is_in_str && !is_in_rgb {
final_str.push(chars[last..index].iter().collect());
last = index + 1
};
}
final_str.push(chars[last..].iter().collect());
final_str
}
fn main() {
let splitted = splitter(r#"test,test: rgba(5,4,3),"test2, seperated""#);
println!("{splitted:?}");
}

Related

Regex to not allow space between words [duplicate]

I'm trying to write a regular expression to remove white spaces from just the beginning of the word, not after, and only a single space after the word.
Used RegExp:
var re = new RegExp(/^([a-zA-Z0-9]+\s?)*$/);
Test Exapmle:
1) test[space]ing - Should be allowed
2) testing - Should be allowed
3) [space]testing - Should not be allowed
4) testing[space] - Should be allowed but have to trim it
5) testing[space][space] - should be allowed but have to trim it
Only one space should be allowed. Is it possible?
To match, what you need, you can use
var re = /^([a-zA-Z0-9]+\s)*[a-zA-Z0-9]+$/;
Maybe you could shorten that a bit, but it matches _ as well
var re = /^(\w+\s)*\w+$/;
function validate(s) {
if (/^(\w+\s?)*\s*$/.test(s)) {
return s.replace(/\s+$/, '');
}
return 'NOT ALLOWED';
}
validate('test ing') // => 'test ing'
validate('testing') // => 'testing'
validate(' testing') // => 'NOT ALLOWED'
validate('testing ') // => 'testing'
validate('testing ') // => 'testing'
validate('test ing ') // => 'test ing'
BTW, new RegExp(..) is redundant if you use regular expression literal.
This one does not allow preceding and following spaces plus only one space between words. Feel free to add any special characters You want.
^([A-Za-z]+ )+[A-Za-z]+$|^[A-Za-z]+$
demo here
Working code- Inside my name.addTextChangedListener():
public void onTextChanged(CharSequence s, int start, int before, int count) {
String n = name.getText().toString();
if (n.equals(""))
name.setError("Name required");
else if (!n.matches("[\\p{Alpha}\\s]*\\b") | n.matches(".*\\s{2}.*") | n.matches("\\s.*")) {
if (n.matches("\\s.*"))
name.setError("Name cannot begin with a space");
else if (n.matches(".*\\s{2}.*"))
name.setError("Multiple spaces between texts");
else if (n.matches(".*\\s"))
name.setError("Blank space at the end of text");
else
name.setError("Non-alphabetic character entered");
}
}
You could try adapting this to your code.
var f=function(t){return Math.pow(t.split(' ').length,2)/t.trim().split(' ').length==2}
f("a a")
true
f("a a ")
false
f("a a")
false
f(" a a")
false
f("a a a")
false
Here is a solution without regular expression.
Add this script inside document.ready function it will work.
var i=0;
jQuery("input,textarea").on('keypress',function(e){
//alert();
if(jQuery(this).val().length < 1){
if(e.which == 32){
//alert(e.which);
return false;
}
}
else {
if(e.which == 32){
if(i != 0){
return false;
}
i++;
}
else{
i=0;
}
}
});
const handleChangeText = text => {
let lastLetter = text[text.length - 1];
let secondLastLetter = text[text.length - 2];
if (lastLetter === ' ' && secondLastLetter === ' ') {
return;
}
setInputText(text.trim());
};
use this
^([A-Za-z]{5,}|[\s]{1}[A-Za-z]{1,})*$
Demo:-https://regex101.com/r/3HP7hl/2

Hive regex for extracting keywords from url

Filenames are following :
file:///storage/emulated/0/SHAREit/videos/Dangerous_Hero_(2017)____Latest_South_Indian_Full_Hindi_Dubbed_Movie___2017_.mp4
file:///storage/emulated/0/VidMate/download/%E0%A0_-_Promo_Songs_-_Khiladi_-_Khesari_Lal_-_Bho.mp4
file:///storage/emulated/0/WhatsApp/Media/WhatsApp%20Video/VID-20171222-WA0015.mp4
file:///storage/emulated/0/bluetooth/%5DChitaChola%7B%7D%D8%B9%D8%A7%D9%85%D8%B1%24%20.3gp
I want to write hive regex to extract words from each string.
for example in 1st string output should be : storage,emulated,....
UPDATE
This Code gives me result , but i wanted regex instead of below code.
package uri_keyword_extractor;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
import java.util.ArrayList;
public class UDFUrlKeywordExtractor extends UDF {
private Text result = new Text();
public Text evaluate(Text url) {
if (url == null) {
return null;
}
String keywords = url_keyword_maker(url.toString());
result.set(keywords);
return result;
}
private static String url_keyword_maker(String url) {
// TODO Auto-generated method stub
ArrayList<String> keywordAr = new ArrayList<String>();
char[] charAr = url.toCharArray();
for (int i = 0; i < charAr.length; i++) {
int current_index = i;
// check if character is a-z or A-Z
char ch = charAr[i];
StringBuilder sb = new StringBuilder();
while (current_index < charAr.length-1 && isChar(ch)) {
sb.append(ch);
current_index = current_index+1;
ch = charAr[current_index];
}
String word = sb.toString();
if (word.length() >= 2) {
keywordAr.add(word);
}
i = current_index;
}
//
StringBuilder sb = new StringBuilder();
for(int i =0; i < keywordAr.size();i++) {
String current = keywordAr.get(i);
sb.append(current);
if(i < keywordAr.size() -1) {
sb.append(",");
}
}
return sb.toString();
}
private static boolean isChar(char ch) {
// TODO Auto-generated method stub
int ascii_value = (int) ch;
// A-Z => (65,90) a-z => (97,122)
// condition 1 : A-Z , condition 2 : a-z character check
if ( (ascii_value >= 65 && ascii_value <= 90) || (ascii_value >= 97 && ascii_value <= 122) ) {
return true;
} else {
return false;
}
}
public static void main(String[] args) {
// TODO Auto-generated method stub
String test1 = "file:///storage/emulated/0/SHAREit/videos/Dangerous_Hero_(2017)____Latest_South_Indian_Full_Hindi_Dubbed_Movie___2017_.mp4";
String test2 = "file:///storage/emulated/0/VidMate/download/%E0%A0_-_Promo_Songs_-_Khiladi_-_Khesari_Lal_-_Bho.mp4";
String test3 = "file:///storage/emulated/0/bluetooth/%5DChitaChola%7B%7D%D8%B9%D8%A7%D9%85%D8%B1%24%20.3gp";
System.out.println(url_keyword_maker(test1).toString());
System.out.println(url_keyword_maker(test2).toString());
System.out.println(url_keyword_maker(test3).toString());
}
}
Use split(str, regex_pattern) function, it splits str using regex as delimiter pattern and returns array. Then use lateral view + epxlode to explode array and filter keywords by length as in your Java code. Then apply collect_set to re-assemble array of keywords+concat_ws(delimeter, str) function to convert array to the delimited string if necessary.
The regex I passed to the split function is '[^a-zA-Z]'.
Demo:
select url_nbr, concat_ws(',',collect_set(key_word)) keywords from
(--your URLs example, url_nbr here is just for reference
select 'file:///storage/emulated/0/SHAREit/videos/Dangerous_Hero_(2017)____Latest_South_Indian_Full_Hindi_Dubbed_Movie___2017_.mp4' as url, 1 as url_nbr union all
select 'file:///storage/emulated/0/VidMate/download/%E0%A0_-_Promo_Songs_-_Khiladi_-_Khesari_Lal_-_Bho.mp4' as url, 2 as url_nbr union all
select 'file:///storage/emulated/0/WhatsApp/Media/WhatsApp%20Video/VID-20171222-WA0015.mp4' as url, 3 as url_nbr union all
select 'file:///storage/emulated/0/bluetooth/%5DChitaChola%7B%7D%D8%B9%D8%A7%D9%85%D8%B1%24%20.3gp' as url, 4 as url_nbr)s
lateral view explode(split(url, '[^a-zA-Z]')) v as key_word
where length(key_word)>=2 --filter here
group by url_nbr
;
Output:
OK
1 file,storage,emulated,SHAREit,videos,Dangerous,Hero,Latest,South,Indian,Full,Hindi,Dubbed,Movie,mp
2 file,storage,emulated,VidMate,download,Promo,Songs,Khiladi,Khesari,Lal,Bho,mp
3 file,storage,emulated,WhatsApp,Media,Video,VID,WA,mp
4 file,storage,emulated,bluetooth,DChitaChola,gp
Time taken: 37.767 seconds, Fetched: 4 row(s)
Maybe I have missed something from your java code, but hope you have caught the idea, so you can easily modify my code and add additional processing if necessary.

How to return only alphanumeric substring?

I'm new to Swift Programming. I would like to ask if anyone can help me to return only alphanumeric substring from a string ?
Example:
Input = "wolf & lion"
Output = "wolflion"
I wonder if there is any solution besides regex.
Thank you
try this:
let outputStr = "wolf & lion".components(separatedBy: CharacterSet.alphanumerics.inverted)
.joined()
print(outputStr)//wolflion
var a = "abs1 2csd^!#awerqwe"
let b = a.characters.map { (char) -> String in
if let charRange = String(char).rangeOfCharacter(from: CharacterSet.alphanumerics) {
return String(char)
} else {
return ""
}
}.joined()
//OR use unicode scalar
let c = a.unicodeScalars.map { (char) -> String in
if CharacterSet.alphanumerics.contains(char) {
return String(char)
} else {
return ""
}
}.joined()
Output: abs12csdawerqwe
Please check :
let str = "wolf & lion"
let charset = str.trimmingCharacters(in: CharacterSet.alphanumerics)
let alphanumericString = str.components(separatedBy: charset).joined(separator: "")
print(alphanumericString) // wolflion

Generate all matches from a subset of regex

I need to define a bunch of vector sequences, which are all a series of L,D,R,U for left, down, right, up or x for break. There are optional parts, and either/or parts. I have been using my own invented system for noting it down, but I want to document this for other, potentially non-programmers to read.
I now want to use a subset (I don't plan on using any wildcards, or infinite repetition for example) of regex to define the vector sequence and a script to produce all possible matching strings...
/LDR/ produces ['LDR']
/LDU?R/ produces ['LDR','LDUR']
/R(LD|DR)U/ produces ['RLDU','RDRU']
/DxR[DL]U?RDRU?/ produces ['DxRDRDR','DxRDRDRU','DxRDURDR','DxRDURDRU','DxRLRDR','DxRLRDRU','DxRLURDR','DxRLURDRU']
Is there an existing library I can use to generate all matches?
EDIT
I realised I will only be needing or statements, as optional things can be specified by thing or nothing maybe a, or b, both optional could be (a|b|). Is there another language I could use to define what I am trying to do?
By translating the java code form the link provided by #Dukeling into javascript, I think I have solved my problem...
var Node = function(str){
this.bracket = false;
this.children = [];
this.s = str;
this.next = null;
this.addChild = function(child){
this.children.push(child);
}
}
var printTree = function(root,prefix){
prefix = prefix.replace(/\./g, "");
for(i in root.children){
var child = root.children[i]
printTree(child, prefix + root.s);
}
if(root.children.length < 1){
console.log(prefix + root.s);
}
}
var Stack = function(){
this.arr = []
this.push = function(item){
this.arr.push(item)
}
this.pop = function(){
return this.arr.pop()
}
this.peek = function(){
return this.arr[this.arr.length-1]
}
}
var createTree = function(s){
// this line was causing errors for `a(((b|c)d)e)f` because the `(((` was only
// replacing the forst two brackets.
// var s = s.replace(/(\(|\||\))(\(|\||\))/g, "$1.$2");
// this line fixes it
var s = s.replace(/[(|)]+/g, function(x){ return x.split('').join('.') });
var str = s.split('');
var stack = new Stack();
var root = new Node("");
stack.push(root); // start node
var justFinishedBrackets = false;
for(i in str){
var c = str[i]
if(c == '('){
stack.peek().next = new Node("Y"); // node after brackets
stack.peek().bracket = true; // node before brackets
} else if (c == '|' || c == ')'){
var last = stack.peek(); // for (ab|cd)e, remember b / d so we can add child e to it
while (!stack.peek().bracket){ // while not node before brackets
stack.pop();
}
last.addChild(stack.peek().next); // for (b|c)d, add d as child to b / c
} else {
if (justFinishedBrackets){
var next = stack.pop().next;
next.s = "" + c;
stack.push(next);
} else {
var n = new Node(""+c);
stack.peek().addChild(n);
stack.push(n);
}
}
justFinishedBrackets = (c == ')');
}
return root;
}
// Test it out
var str = "a(c|mo(r|l))e";
var root = createTree(str);
printTree(root, "");
// Prints: ace / amore / amole
I only changed one line, to allow more than two consecutive brackets to be handled, and left the original translation in the comments
I also added a function to return an array of results, instead of printing them...
var getTree = function(root,prefix){
this.out = this.out || []
prefix = prefix.replace(/\./g, "");
for(i in root.children){
var child = root.children[i]
getTree(child, prefix + root.s, out);
}
if(root.children.length < 1){
this.out.push(prefix + root.s);
}
if(!prefix && !root.s){
var out = this.out;
this.out = null
return out;
}
}
// Test it
var str = "a(b|c)d";
var root = createTree(str);
console.log(getTree(root, ""));
// logs ["abd","acd"]
The last part, to allow for empty strings too, so... (ab|c|) means ab or c or nothing, and a convenience shortcut so that ab?c is translated into a(b|)c.
var getMatches = function(str){
str = str.replace(/(.)\?/g,"($1|)")
// replace all instances of `(???|)` with `(???|µ)`
// the µ will be stripped out later
str = str.replace(/\|\)/g,"|µ)")
// fix issues where last character is `)` by inserting token `µ`
// which will be stripped out later
str = str+"µ"
var root = createTree(str);
var res = getTree(root, "");
// strip out token µ
for(i in res){
res[i] = res[i].replace(/µ/g,"")
}
// return the array of results
return res
}
getMatches("a(bc|de?)?f");
// Returns: ["abcf","adef","adf","af"]
The last part is a little hack-ish as it relies on µ not being in the string (not an issue for me) and solves one bug, where a ) at the end on the input string was causing incorrect output, by inserting a µ at the end of each string, and then stripping it from the results. I would be happy for someone to suggest a better way to handle these issues, so it can work as a more general solution.
This code as it stands does everything I need. Thanks for all your help!
I'd imagine what you're trying is quite easy with a tree (as long as it's only or-statements).
Parse a(b|c)d (or any or-statement) into a tree as follows: a has children b and c, b and c have a mutual child d. b and c can both consist of 0 or more nodes (as in c could be g(e|f)h in which case (part of) the tree would be a -> g -> e/f (2 nodes) -> h -> d or c could be empty, in which case (part of) the tree would be a -> d, but an actual physical empty node may simplify things, which you should see when trying to write the code).
Generation of the tree shouldn't be too difficult with either recursion or a stack.
Once you have a tree, it's trivial to recursively iterate through the whole thing and generate all strings.
Also, here is a link to a similar question, providing a library or two.
EDIT:
"shouldn't be too difficult" - okay, maybe not
Here is a somewhat complicated example (Java) that may require some advanced knowledge about stacks.
Here is a slightly simpler version (Java) thanks to inserting a special character between each ((, )), |(, etc.
Note that neither of these are particularly efficient, the point is just to get the idea across.
Here is a JavaScript example that addresses parsing the (a|b) and (a|b|) possibilities, creates an array of possible substrings, and composes the matches based on this answer.
var regex = /\([RLUD]*\|[RLUD]*\|?\)/,
str = "R(LD|DR)U(R|L|)",
substrings = [], matches = [], str_tmp = str, find
while (find = regex.exec(str_tmp)){
var index = find.index
finds = find[0].split(/\|/)
substrings.push(str_tmp.substr(0, index))
if (find[0].match(/\|/g).length == 1)
substrings.push([finds[0].substr(1), finds[1].replace(/.$/, '')])
else if (find[0].match(/\|/g).length == 2){
substrings.push([finds[0].substr(1), ""])
substrings.push([finds[1], ""])
}
str_tmp = str_tmp.substr(index + find[0].length)
}
if (str_tmp) substrings.push([str_tmp])
console.log(substrings) //>>["R", ["LD", "DR"], "U", ["R", ""], ["L", ""]]
//compose matches
function printBin(tree, soFar, iterations) {
if (iterations == tree.length) matches.push(soFar)
else if (tree[iterations].length == 2){
printBin(tree, soFar + tree[iterations][0], iterations + 1)
printBin(tree, soFar + tree[iterations][1], iterations + 1)
}
else printBin(tree, soFar + tree[iterations], iterations + 1)
}
printBin(substrings, "", 0)
console.log(matches) //>>["RLDURL", "RLDUR", "RLDUL", "RLDU", "RDRURL", "RDRUR", "RDRUL", "RDRU"]

How does Stack Overflow generate its SEO-friendly URLs?

What is a good complete regular expression or some other process that would take the title:
How do you change a title to be part of the URL like Stack Overflow?
and turn it into
how-do-you-change-a-title-to-be-part-of-the-url-like-stack-overflow
that is used in the SEO-friendly URLs on Stack Overflow?
The development environment I am using is Ruby on Rails, but if there are some other platform-specific solutions (.NET, PHP, Django), I would love to see those too.
I am sure I (or another reader) will come across the same problem on a different platform down the line.
I am using custom routes, and I mainly want to know how to alter the string to all special characters are removed, it's all lowercase, and all whitespace is replaced.
Here's how we do it. Note that there are probably more edge conditions than you realize at first glance.
This is the second version, unrolled for 5x more performance (and yes, I benchmarked it). I figured I'd optimize it because this function can be called hundreds of times per page.
/// <summary>
/// Produces optional, URL-friendly version of a title, "like-this-one".
/// hand-tuned for speed, reflects performance refactoring contributed
/// by John Gietzen (user otac0n)
/// </summary>
public static string URLFriendly(string title)
{
if (title == null) return "";
const int maxlen = 80;
int len = title.Length;
bool prevdash = false;
var sb = new StringBuilder(len);
char c;
for (int i = 0; i < len; i++)
{
c = title[i];
if ((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9'))
{
sb.Append(c);
prevdash = false;
}
else if (c >= 'A' && c <= 'Z')
{
// tricky way to convert to lowercase
sb.Append((char)(c | 32));
prevdash = false;
}
else if (c == ' ' || c == ',' || c == '.' || c == '/' ||
c == '\\' || c == '-' || c == '_' || c == '=')
{
if (!prevdash && sb.Length > 0)
{
sb.Append('-');
prevdash = true;
}
}
else if ((int)c >= 128)
{
int prevlen = sb.Length;
sb.Append(RemapInternationalCharToAscii(c));
if (prevlen != sb.Length) prevdash = false;
}
if (i == maxlen) break;
}
if (prevdash)
return sb.ToString().Substring(0, sb.Length - 1);
else
return sb.ToString();
}
To see the previous version of the code this replaced (but is functionally equivalent to, and 5x faster), view revision history of this post (click the date link).
Also, the RemapInternationalCharToAscii method source code can be found here.
Here is my version of Jeff's code. I've made the following changes:
The hyphens were appended in such a way that one could be added, and then need removing as it was the last character in the string. That is, we never want “my-slug-”. This means an extra string allocation to remove it on this edge case. I’ve worked around this by delay-hyphening. If you compare my code to Jeff’s the logic for this is easy to follow.
His approach is purely lookup based and missed a lot of characters I found in examples while researching on Stack Overflow. To counter this, I first peform a normalisation pass (AKA collation mentioned in Meta Stack Overflow question Non US-ASCII characters dropped from full (profile) URL), and then ignore any characters outside the acceptable ranges. This works most of the time...
... For when it doesn’t I’ve also had to add a lookup table. As mentioned above, some characters don’t map to a low ASCII value when normalised. Rather than drop these I’ve got a manual list of exceptions that is doubtless full of holes, but it is better than nothing. The normalisation code was inspired by Jon Hanna’s great post in Stack Overflow question How can I remove accents on a string?.
The case conversion is now also optional.
public static class Slug
{
public static string Create(bool toLower, params string[] values)
{
return Create(toLower, String.Join("-", values));
}
/// <summary>
/// Creates a slug.
/// References:
/// http://www.unicode.org/reports/tr15/tr15-34.html
/// https://meta.stackexchange.com/questions/7435/non-us-ascii-characters-dropped-from-full-profile-url/7696#7696
/// https://stackoverflow.com/questions/25259/how-do-you-include-a-webpage-title-as-part-of-a-webpage-url/25486#25486
/// https://stackoverflow.com/questions/3769457/how-can-i-remove-accents-on-a-string
/// </summary>
/// <param name="toLower"></param>
/// <param name="normalised"></param>
/// <returns></returns>
public static string Create(bool toLower, string value)
{
if (value == null)
return "";
var normalised = value.Normalize(NormalizationForm.FormKD);
const int maxlen = 80;
int len = normalised.Length;
bool prevDash = false;
var sb = new StringBuilder(len);
char c;
for (int i = 0; i < len; i++)
{
c = normalised[i];
if ((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9'))
{
if (prevDash)
{
sb.Append('-');
prevDash = false;
}
sb.Append(c);
}
else if (c >= 'A' && c <= 'Z')
{
if (prevDash)
{
sb.Append('-');
prevDash = false;
}
// Tricky way to convert to lowercase
if (toLower)
sb.Append((char)(c | 32));
else
sb.Append(c);
}
else if (c == ' ' || c == ',' || c == '.' || c == '/' || c == '\\' || c == '-' || c == '_' || c == '=')
{
if (!prevDash && sb.Length > 0)
{
prevDash = true;
}
}
else
{
string swap = ConvertEdgeCases(c, toLower);
if (swap != null)
{
if (prevDash)
{
sb.Append('-');
prevDash = false;
}
sb.Append(swap);
}
}
if (sb.Length == maxlen)
break;
}
return sb.ToString();
}
static string ConvertEdgeCases(char c, bool toLower)
{
string swap = null;
switch (c)
{
case 'ı':
swap = "i";
break;
case 'ł':
swap = "l";
break;
case 'Ł':
swap = toLower ? "l" : "L";
break;
case 'đ':
swap = "d";
break;
case 'ß':
swap = "ss";
break;
case 'ø':
swap = "o";
break;
case 'Þ':
swap = "th";
break;
}
return swap;
}
}
For more details, the unit tests, and an explanation of why Facebook's URL scheme is a little smarter than Stack Overflows, I've got an expanded version of this on my blog.
You will want to setup a custom route to point the URL to the controller that will handle it. Since you are using Ruby on Rails, here is an introduction in using their routing engine.
In Ruby, you will need a regular expression like you already know and here is the regular expression to use:
def permalink_for(str)
str.gsub(/[^\w\/]|[!\(\)\.]+/, ' ').strip.downcase.gsub(/\ +/, '-')
end
You can also use this JavaScript function for in-form generation of the slug's (this one is based on/copied from Django):
function makeSlug(urlString, filter) {
// Changes, e.g., "Petty theft" to "petty_theft".
// Remove all these words from the string before URLifying
if(filter) {
removelist = ["a", "an", "as", "at", "before", "but", "by", "for", "from",
"is", "in", "into", "like", "of", "off", "on", "onto", "per",
"since", "than", "the", "this", "that", "to", "up", "via", "het", "de", "een", "en",
"with"];
}
else {
removelist = [];
}
s = urlString;
r = new RegExp('\\b(' + removelist.join('|') + ')\\b', 'gi');
s = s.replace(r, '');
s = s.replace(/[^-\w\s]/g, ''); // Remove unneeded characters
s = s.replace(/^\s+|\s+$/g, ''); // Trim leading/trailing spaces
s = s.replace(/[-\s]+/g, '-'); // Convert spaces to hyphens
s = s.toLowerCase(); // Convert to lowercase
return s; // Trim to first num_chars characters
}
For good measure, here's the PHP function in WordPress that does it... I'd think that WordPress is one of the more popular platforms that uses fancy links.
function sanitize_title_with_dashes($title) {
$title = strip_tags($title);
// Preserve escaped octets.
$title = preg_replace('|%([a-fA-F0-9][a-fA-F0-9])|', '---$1---', $title);
// Remove percent signs that are not part of an octet.
$title = str_replace('%', '', $title);
// Restore octets.
$title = preg_replace('|---([a-fA-F0-9][a-fA-F0-9])---|', '%$1', $title);
$title = remove_accents($title);
if (seems_utf8($title)) {
if (function_exists('mb_strtolower')) {
$title = mb_strtolower($title, 'UTF-8');
}
$title = utf8_uri_encode($title, 200);
}
$title = strtolower($title);
$title = preg_replace('/&.+?;/', '', $title); // kill entities
$title = preg_replace('/[^%a-z0-9 _-]/', '', $title);
$title = preg_replace('/\s+/', '-', $title);
$title = preg_replace('|-+|', '-', $title);
$title = trim($title, '-');
return $title;
}
This function as well as some of the supporting functions can be found in wp-includes/formatting.php.
If you are using Rails edge, you can rely on Inflector.parametrize - here's the example from the documentation:
class Person
def to_param
"#{id}-#{name.parameterize}"
end
end
#person = Person.find(1)
# => #<Person id: 1, name: "Donald E. Knuth">
<%= link_to(#person.name, person_path(#person)) %>
# => Donald E. Knuth
Also if you need to handle more exotic characters such as accents (éphémère) in previous version of Rails, you can use a mixture of PermalinkFu and DiacriticsFu:
DiacriticsFu::escape("éphémère")
=> "ephemere"
DiacriticsFu::escape("räksmörgås")
=> "raksmorgas"
I am not familiar with Ruby on Rails, but the following is (untested) PHP code. You can probably translate this very quickly to Ruby on Rails if you find it useful.
$sURL = "This is a title to convert to URL-format. It has 1 number in it!";
// To lower-case
$sURL = strtolower($sURL);
// Replace all non-word characters with spaces
$sURL = preg_replace("/\W+/", " ", $sURL);
// Remove trailing spaces (so we won't end with a separator)
$sURL = trim($sURL);
// Replace spaces with separators (hyphens)
$sURL = str_replace(" ", "-", $sURL);
echo $sURL;
// outputs: this-is-a-title-to-convert-to-url-format-it-has-1-number-in-it
I hope this helps.
I don't much about Ruby or Rails, but in Perl, this is what I would do:
my $title = "How do you change a title to be part of the url like Stackoverflow?";
my $url = lc $title; # Change to lower case and copy to URL.
$url =~ s/^\s+//g; # Remove leading spaces.
$url =~ s/\s+$//g; # Remove trailing spaces.
$url =~ s/\s+/\-/g; # Change one or more spaces to single hyphen.
$url =~ s/[^\w\-]//g; # Remove any non-word characters.
print "$title\n$url\n";
I just did a quick test and it seems to work. Hopefully this is relatively easy to translate to Ruby.
T-SQL implementation, adapted from dbo.UrlEncode:
CREATE FUNCTION dbo.Slug(#string varchar(1024))
RETURNS varchar(3072)
AS
BEGIN
DECLARE #count int, #c char(1), #i int, #slug varchar(3072)
SET #string = replace(lower(ltrim(rtrim(#string))),' ','-')
SET #count = Len(#string)
SET #i = 1
SET #slug = ''
WHILE (#i <= #count)
BEGIN
SET #c = substring(#string, #i, 1)
IF #c LIKE '[a-z0-9--]'
SET #slug = #slug + #c
SET #i = #i +1
END
RETURN #slug
END
I know it's very old question but since most of the browsers now support unicode urls I found a great solution in XRegex that converts everything except letters (in all languages to '-').
That can be done in several programming languages.
The pattern is \\p{^L}+ and then you just need to use it to replace all non letters to '-'.
Working example in node.js with xregex module.
var text = 'This ! can # have # several $ letters % from different languages such as עברית or Español';
var slugRegEx = XRegExp('((?!\\d)\\p{^L})+', 'g');
var slug = XRegExp.replace(text, slugRegEx, '-').toLowerCase();
console.log(slug) ==> "this-can-have-several-letters-from-different-languages-such-as-עברית-or-español"
Assuming that your model class has a title attribute, you can simply override the to_param method within the model, like this:
def to_param
title.downcase.gsub(/ /, '-')
end
This Railscast episode has all the details. You can also ensure that the title only contains valid characters using this:
validates_format_of :title, :with => /^[a-z0-9-]+$/,
:message => 'can only contain letters, numbers and hyphens'
Brian's code, in Ruby:
title.downcase.strip.gsub(/\ /, '-').gsub(/[^\w\-]/, '')
downcase turns the string to lowercase, strip removes leading and trailing whitespace, the first gsub call globally substitutes spaces with dashes, and the second removes everything that isn't a letter or a dash.
There is a small Ruby on Rails plugin called PermalinkFu, that does this. The escape method does the transformation into a string that is suitable for a URL. Have a look at the code; that method is quite simple.
To remove non-ASCII characters it uses the iconv lib to translate to 'ascii//ignore//translit' from 'utf-8'. Spaces are then turned into dashes, everything is downcased, etc.
You can use the following helper method. It can convert the Unicode characters.
public static string ConvertTextToSlug(string s)
{
StringBuilder sb = new StringBuilder();
bool wasHyphen = true;
foreach (char c in s)
{
if (char.IsLetterOrDigit(c))
{
sb.Append(char.ToLower(c));
wasHyphen = false;
}
else
if (char.IsWhiteSpace(c) && !wasHyphen)
{
sb.Append('-');
wasHyphen = true;
}
}
// Avoid trailing hyphens
if (wasHyphen && sb.Length > 0)
sb.Length--;
return sb.ToString().Replace("--","-");
}
Here's my (slower, but fun to write) version of Jeff's code:
public static string URLFriendly(string title)
{
char? prevRead = null,
prevWritten = null;
var seq =
from c in title
let norm = RemapInternationalCharToAscii(char.ToLowerInvariant(c).ToString())[0]
let keep = char.IsLetterOrDigit(norm)
where prevRead.HasValue || keep
let replaced = keep ? norm
: prevWritten != '-' ? '-'
: (char?)null
where replaced != null
let s = replaced + (prevRead == null ? ""
: norm == '#' && "cf".Contains(prevRead.Value) ? "sharp"
: norm == '+' ? "plus"
: "")
let _ = prevRead = norm
from written in s
let __ = prevWritten = written
select written;
const int maxlen = 80;
return string.Concat(seq.Take(maxlen)).TrimEnd('-');
}
public static string RemapInternationalCharToAscii(string text)
{
var seq = text.Normalize(NormalizationForm.FormD)
.Where(c => CharUnicodeInfo.GetUnicodeCategory(c) != UnicodeCategory.NonSpacingMark);
return string.Concat(seq).Normalize(NormalizationForm.FormC);
}
My test string:
" I love C#, F#, C++, and... Crème brûlée!!! They see me codin'... they hatin'... tryin' to catch me codin' dirty... "
The stackoverflow solution is great, but modern browser (excluding IE, as usual) now handle nicely utf8 encoding:
So I upgraded the proposed solution:
public static string ToFriendlyUrl(string title, bool useUTF8Encoding = false)
{
...
else if (c >= 128)
{
int prevlen = sb.Length;
if (useUTF8Encoding )
{
sb.Append(HttpUtility.UrlEncode(c.ToString(CultureInfo.InvariantCulture),Encoding.UTF8));
}
else
{
sb.Append(RemapInternationalCharToAscii(c));
}
...
}
Full Code on Pastebin
Edit: Here's the code for RemapInternationalCharToAscii method (that's missing in the pastebin).
I liked the way this is done without using regular expressions, so I ported it to PHP. I just added a function called is_between to check characters:
function is_between($val, $min, $max)
{
$val = (int) $val; $min = (int) $min; $max = (int) $max;
return ($val >= $min && $val <= $max);
}
function international_char_to_ascii($char)
{
if (mb_strpos('àåáâäãåa', $char) !== false)
{
return 'a';
}
if (mb_strpos('èéêëe', $char) !== false)
{
return 'e';
}
if (mb_strpos('ìíîïi', $char) !== false)
{
return 'i';
}
if (mb_strpos('òóôõö', $char) !== false)
{
return 'o';
}
if (mb_strpos('ùúûüuu', $char) !== false)
{
return 'u';
}
if (mb_strpos('çccc', $char) !== false)
{
return 'c';
}
if (mb_strpos('zzž', $char) !== false)
{
return 'z';
}
if (mb_strpos('ssšs', $char) !== false)
{
return 's';
}
if (mb_strpos('ñn', $char) !== false)
{
return 'n';
}
if (mb_strpos('ýÿ', $char) !== false)
{
return 'y';
}
if (mb_strpos('gg', $char) !== false)
{
return 'g';
}
if (mb_strpos('r', $char) !== false)
{
return 'r';
}
if (mb_strpos('l', $char) !== false)
{
return 'l';
}
if (mb_strpos('d', $char) !== false)
{
return 'd';
}
if (mb_strpos('ß', $char) !== false)
{
return 'ss';
}
if (mb_strpos('Þ', $char) !== false)
{
return 'th';
}
if (mb_strpos('h', $char) !== false)
{
return 'h';
}
if (mb_strpos('j', $char) !== false)
{
return 'j';
}
return '';
}
function url_friendly_title($url_title)
{
if (empty($url_title))
{
return '';
}
$url_title = mb_strtolower($url_title);
$url_title_max_length = 80;
$url_title_length = mb_strlen($url_title);
$url_title_friendly = '';
$url_title_dash_added = false;
$url_title_char = '';
for ($i = 0; $i < $url_title_length; $i++)
{
$url_title_char = mb_substr($url_title, $i, 1);
if (strlen($url_title_char) == 2)
{
$url_title_ascii = ord($url_title_char[0]) * 256 + ord($url_title_char[1]) . "\r\n";
}
else
{
$url_title_ascii = ord($url_title_char);
}
if (is_between($url_title_ascii, 97, 122) || is_between($url_title_ascii, 48, 57))
{
$url_title_friendly .= $url_title_char;
$url_title_dash_added = false;
}
elseif(is_between($url_title_ascii, 65, 90))
{
$url_title_friendly .= chr(($url_title_ascii | 32));
$url_title_dash_added = false;
}
elseif($url_title_ascii == 32 || $url_title_ascii == 44 || $url_title_ascii == 46 || $url_title_ascii == 47 || $url_title_ascii == 92 || $url_title_ascii == 45 || $url_title_ascii == 47 || $url_title_ascii == 95 || $url_title_ascii == 61)
{
if (!$url_title_dash_added && mb_strlen($url_title_friendly) > 0)
{
$url_title_friendly .= chr(45);
$url_title_dash_added = true;
}
}
else if ($url_title_ascii >= 128)
{
$url_title_previous_length = mb_strlen($url_title_friendly);
$url_title_friendly .= international_char_to_ascii($url_title_char);
if ($url_title_previous_length != mb_strlen($url_title_friendly))
{
$url_title_dash_added = false;
}
}
if ($i == $url_title_max_length)
{
break;
}
}
if ($url_title_dash_added)
{
return mb_substr($url_title_friendly, 0, -1);
}
else
{
return $url_title_friendly;
}
}
Now all Browser handle nicely utf8 encoding, so you can use WebUtility.UrlEncode Method , its like HttpUtility.UrlEncode used by #giamin but its work outside of a web application.
I ported the code to TypeScript. It can easily be adapted to JavaScript.
I am adding a .contains method to the String prototype, if you're targeting the latest browsers or ES6 you can use .includes instead.
if (!String.prototype.contains) {
String.prototype.contains = function (check) {
return this.indexOf(check, 0) !== -1;
};
}
declare interface String {
contains(check: string): boolean;
}
export function MakeUrlFriendly(title: string) {
if (title == null || title == '')
return '';
const maxlen = 80;
let len = title.length;
let prevdash = false;
let result = '';
let c: string;
let cc: number;
let remapInternationalCharToAscii = function (c: string) {
let s = c.toLowerCase();
if ("àåáâäãåą".contains(s)) {
return "a";
}
else if ("èéêëę".contains(s)) {
return "e";
}
else if ("ìíîïı".contains(s)) {
return "i";
}
else if ("òóôõöøőð".contains(s)) {
return "o";
}
else if ("ùúûüŭů".contains(s)) {
return "u";
}
else if ("çćčĉ".contains(s)) {
return "c";
}
else if ("żźž".contains(s)) {
return "z";
}
else if ("śşšŝ".contains(s)) {
return "s";
}
else if ("ñń".contains(s)) {
return "n";
}
else if ("ýÿ".contains(s)) {
return "y";
}
else if ("ğĝ".contains(s)) {
return "g";
}
else if (c == 'ř') {
return "r";
}
else if (c == 'ł') {
return "l";
}
else if (c == 'đ') {
return "d";
}
else if (c == 'ß') {
return "ss";
}
else if (c == 'Þ') {
return "th";
}
else if (c == 'ĥ') {
return "h";
}
else if (c == 'ĵ') {
return "j";
}
else {
return "";
}
};
for (let i = 0; i < len; i++) {
c = title[i];
cc = c.charCodeAt(0);
if ((cc >= 97 /* a */ && cc <= 122 /* z */) || (cc >= 48 /* 0 */ && cc <= 57 /* 9 */)) {
result += c;
prevdash = false;
}
else if ((cc >= 65 && cc <= 90 /* A - Z */)) {
result += c.toLowerCase();
prevdash = false;
}
else if (c == ' ' || c == ',' || c == '.' || c == '/' || c == '\\' || c == '-' || c == '_' || c == '=') {
if (!prevdash && result.length > 0) {
result += '-';
prevdash = true;
}
}
else if (cc >= 128) {
let prevlen = result.length;
result += remapInternationalCharToAscii(c);
if (prevlen != result.length) prevdash = false;
}
if (i == maxlen) break;
}
if (prevdash)
return result.substring(0, result.length - 1);
else
return result;
}
No, no, no. You are all so very wrong. Except for the diacritics-fu stuff, you're getting there, but what about Asian characters (shame on Ruby developers for not considering their nihonjin brethren).
Firefox and Safari both display non-ASCII characters in the URL, and frankly they look great. It is nice to support links like 'http://somewhere.com/news/read/お前たちはアホじゃないかい'.
So here's some PHP code that'll do it, but I just wrote it and haven't stress tested it.
<?php
function slug($str)
{
$args = func_get_args();
array_filter($args); //remove blanks
$slug = mb_strtolower(implode('-', $args));
$real_slug = '';
$hyphen = '';
foreach(SU::mb_str_split($slug) as $c)
{
if (strlen($c) > 1 && mb_strlen($c)===1)
{
$real_slug .= $hyphen . $c;
$hyphen = '';
}
else
{
switch($c)
{
case '&':
$hyphen = $real_slug ? '-and-' : '';
break;
case 'a':
case 'b':
case 'c':
case 'd':
case 'e':
case 'f':
case 'g':
case 'h':
case 'i':
case 'j':
case 'k':
case 'l':
case 'm':
case 'n':
case 'o':
case 'p':
case 'q':
case 'r':
case 's':
case 't':
case 'u':
case 'v':
case 'w':
case 'x':
case 'y':
case 'z':
case 'A':
case 'B':
case 'C':
case 'D':
case 'E':
case 'F':
case 'G':
case 'H':
case 'I':
case 'J':
case 'K':
case 'L':
case 'M':
case 'N':
case 'O':
case 'P':
case 'Q':
case 'R':
case 'S':
case 'T':
case 'U':
case 'V':
case 'W':
case 'X':
case 'Y':
case 'Z':
case '0':
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9':
$real_slug .= $hyphen . $c;
$hyphen = '';
break;
default:
$hyphen = $hyphen ? $hyphen : ($real_slug ? '-' : '');
}
}
}
return $real_slug;
}
Example:
$str = "~!##$%^&*()_+-=[]\{}|;':\",./<>?\n\r\t\x07\x00\x04 コリン ~!##$%^&*()_+-=[]\{}|;':\",./<>?\n\r\t\x07\x00\x04 トーマス ~!##$%^&*()_+-=[]\{}|;':\",./<>?\n\r\t\x07\x00\x04 アーノルド ~!##$%^&*()_+-=[]\{}|;':\",./<>?\n\r\t\x07\x00\x04";
echo slug($str);
Outputs:
コリン-and-トーマス-and-アーノルド
The '-and-' is because &'s get changed to '-and-'.
Rewrite of Jeff's code to be more concise
public static string RemapInternationalCharToAscii(char c)
{
var s = c.ToString().ToLowerInvariant();
var mappings = new Dictionary<string, string>
{
{ "a", "àåáâäãåą" },
{ "c", "çćčĉ" },
{ "d", "đ" },
{ "e", "èéêëę" },
{ "g", "ğĝ" },
{ "h", "ĥ" },
{ "i", "ìíîïı" },
{ "j", "ĵ" },
{ "l", "ł" },
{ "n", "ñń" },
{ "o", "òóôõöøőð" },
{ "r", "ř" },
{ "s", "śşšŝ" },
{ "ss", "ß" },
{ "th", "Þ" },
{ "u", "ùúûüŭů" },
{ "y", "ýÿ" },
{ "z", "żźž" }
};
foreach(var mapping in mappings)
{
if (mapping.Value.Contains(s))
return mapping.Key;
}
return string.Empty;
}