Certain UTF-8 symbols break subject lines. I see this problem in konsole (ubuntu) and termux (android) terminals. Bad subject line:
Subject: =?utf-8?B?4pqh77iP0JDQutGG0LjRjyDQt9Cw0LLQtdGA0YjQsNC10YLRgdGPISA=?=
=?utf-8?B?0KPRgdC/0LXQudGC0LUg0LfQsNCx0YDQsNGC0Ywg0J7QkdCYINGA0YPQsdC7?=
=?utf-8?B?0Lg=?=
If I place cursor on that line then the termanal looks like this:
And if I walk down and up a bit then it duplicates next line, pulling text:
And everything below that line is messed up. Any ideas how to filter out these characters? Nowadays more and more people use these "fancy" things in subjects, breaking message list.
This problem is common for all versions of mutt and neomutt, old and new.
Related
I have a large text file that I'm going to be working with programmatically but have run into problems with a special character strewn throughout the file. The file is way too large to scan it looking for specific characters. Most of the other unwanted special characters I've been able to get rid of using some regex pattern. But there is a box character, similar to "□". When I tried to copy the character from the actual text file and past it here I get "�", so the example of the box is from Windows character map which includes the code 'U+25A1', which I'm not sure how to interpret or if it's something I could use for a regex search.
Would anyone know how I could search for the box symbol similar to "□" in a UTF-8 encoded file?
EDIT:
Here is an example from the text file:
"� Prune palms when flower spathes show, or delay pruning until after the palm has finished flowering, to prevent infestation of palm flower caterpillars. Leave the top five rows."
The only problem is that, as mentioned in the original post, the square gets converted into a diamond question mark.
It's unclear where and how you are searching, although you could use the hex equivalent:
\x{25A1}
Example:
https://regex101.com/r/b84oBs/1
The black diamond with a question mark is not a character, per se. It is what a browser spits out at you when you give it unrecognizable bytes.
Find out where that data is coming from.
Determine its encoding. (Usually UTF-8, but might be something else.)
Be sure the browser is configured to display that encoding. This is likely to suffice <meta charset=UTF-8> in the header of the page.
I found a workaround using Notepad++ and this website. It's still not clear what encoding system the square is originally from, but when I post it into the query field in the website above or into the Notepad++ Conversion Table (Plugins > Converter > Conversion Table) it gives the hex-character code for the "Replacement Character" which is the diamond with the question mark.
Using this code in a regex expression, \x{FFFD}, within Notepad++ search gave me all the squares, although recognizing them as the Replacement Character.
The time of day that you have missed. Recently I began to study the expressive expressions, but the task before me is too complicated.
I think many people need the ability to quickly format texts of almost any type. Problem is not easy and I hope to get a solution to this problem from professionals.
If you use a limited breakdown for each line, then only large lines are broken which is much better. That is, to break only those lines that are larger than a certain size and which are more or less evenly broken Correct formatting of the text is quite Difficult to take into account a lot, I will go out for a long time. As you can see, there are small lines and they have to be sent back by connecting the previous line, but the problem is. And again have to apply formatting. But it is unclear whether this new formatting will create new problems.
All this is write in the macro via notepad ++ one after another and use.
However, it is necessary to solve the most important problems:
It is important to
I want to immediately note: textFX does not offer. My attempts to write a macro with textFX (including attempts with 75 or a number from the clipboard failed, the text was written with unreadable code), I sent a corresponding message to the site notepad ++ a day ago
Use a regular point also do not offer. The fact is that notepad ++ macro does not understand the point of replacing the indentation with spaces by default 4pcs. (Yes, the macro must also indent the text, but this is all right)
About soft wordwrap (not formatted just click) is also not worth talking about.
inscrease line indent 2-4
split lines
descrease line indet 2-4
^(.?)$\s+?^(?=.^\1$) null
2017 (\d+:\d+) 17
subscribe .+$ null
(^.{1,30}$)\R \1
\r\r\n \n
^ spaces\1
i create macro and solved more problems
Is there a way to autoindent or a shortcut to insert a select number of spaces as a tab in the SAS Progam Editor on Unix?
I'm used to using the enhanced editor on PC and this is the only part of the switch that I can't find the answer to.
On the line numbers to the left of the code in the program editor, enter the >># command (replace # with a number) to indent '#' spaces to the right and <<# for shifting to the left. Enter these commands on the starting line and ending line. All rows inclusive between them will shift.
For a single line, ># or <#.
Without trying to start a flame-war... have you and your team considered NOT using the tab key in code? This way code always appears consistent regardless of editor/tab settings. Especially handy when you use a combination of vi/EG/SAS editor/notepad etc... I've worked at places that do this and it works great once you get everyone to agree to it.
When you open up existing code with tabs, just figure out how many spaces the tab is supposed to represent and do a search/replace.
It's still as fast to navigate quickly when you use, ctrl-left/ctrl-right in windows editors (and unix eds if setup that way), or 'w' and 'b' to jump to the next/previous word when using vi.
Cheers
Rob
I have a peculiar problem. I have an email group that pipes emails to a message board. The word wrap of the emails varies. In yahoo, the messages tend to fill the entire container on the message board. But in all other mail clients, only part of the container width is filled, because the original mail was wrapped. I want all of the email messages to fill the entire width of the container. I've thought of two possible solutions: CSS, or a Regex that eliminates line breaks. Because I am only a garage mechanic (at these sorts of things), I simply cannot get the job done. Any help out there?
Here is a link that shows the issue: http://seanwilson.org/forum/index.php?t=msg&th=1729&start=0&S=171399e41f2c10c4357dd9b217caaa3f
(compare the message of "sean" with that of "rob." One fills the container, the other not).
Can any of you suggest how to get all the mail to fill the container?
You gave too little information - what programming language are you using - PHP/Javascript/anything different?
I think you only need to replace \n, \r and \r\n with whitespace. PHP code for that:
$nowrap = str_replace('\r\n', ' ', $nowrap);
$nowrap = str_replace('\r', ' ', $nowrap);
$nowrap = str_replace('\n', ' ', $nowrap);
You can do that analogically in other languages (for JS see string.replace method: http://www.tizag.com/javascriptT/javascript-string-replace.php).
Depending on the situation (people always seem to add 2 linebreaks between paragraphs), you could say the problem is: replace all newlines not directly preceded or followed by a newline with a space.
//just to be sure, remove \r's
$string = str_replace("\r",'',$string);
$string = preg_replace('/(?<!\n)\n(?!\n)/',' ',$string);
While allowing \r's:
$string = preg_replace('/(?<!\r|\n)\r?\n(?!\r|\n)/',' ',$string);
Edit: nevermind: do not use: while people tend to write their email text in paragraphs, you will break their signature / signoff with this regex. One could fiddle around with a minimum linelength before deeming it 'breakable' (i chose 63), but fiddly it will be:
$string = preg_replace('/([^\r\n]{63,})\r?\n(?!\r|\n)/','$1 ',$string);
The problem is: there are no assurance the linebreak wasn't intended. With a fiddleable line-length you could base it on average users, but the question is: what do they mind more: the differences between breaking & non-breaking paragraphs, or the breaking of their signatures?
Thanks for getting back so quickly!
The discussion board uses php (and also CSS). The only trouble is that I am somewhat limited in my ability to tinker with its programing. If I am to do this at my current level of skilty, I have only one of two options.
using a preg-replace in php. The discussion board allows us to do this from a control panel. So If I could do it with one preg-replace statement, it should work.
Would Wrikken's solution work if I do not remove \r's? Because that seems to be spot on. (could the \r's be added to the preg-replace?)
I had hoped the solution could come through a css property of some sort. I guess that isn't possible.
Thanks so much for your help!
[NOTE: thanks so much for your help! The solution worked!!! I changed the number to 53 or so. It needed to be a little smaller. I don't care that a rare, long signature lines may lose its carriage return. That's a small price to pay for a full message box! You easily saved me several days of learning something that was bound to be moderately frustrating, Thanks so much for that quick fix. I am joyous at the help I received here.]
I'm working on a web application that parses and displays email messages in a threaded format (among other things). Emails may come from any number of different mail clients, and in either text or HTML format.
Given that most people have a tendency to top post, I'd like to be able to hide the duplicated message in an email reply in a manner similar to how Gmail does it (e.g. "show quoted text").
Determining which part of the message is the reply is somewhat challenging. Personally, I use "> " delimiters at the beginning of the quoted text when replying. I created a regexp that looks for these lines and wraps a div around them to allow some JS to hide or show this block of text.
I then noticed that Outlook doesn't use the "> " characters by default, it simply adds a header block above the reply with the summary of the headers (From, Subject, Date, etc.). The reply is untouched. I can match on this and hide the rest of the email, working with the assumption that it's a top quote.
I then looked at Thunderbird, and it uses "> " for text, and <blockquote> for HTML mails. I still haven't looked at what Apple Mail does, what Notes does, or what any of the other millions of mail clients out there do.
Will I be writing a special case regexp for every single client out there? or is there something I'm missing?
Any suggestions, sample code or pointers to third party libraries much appreciated!
It'll be pretty hard to duplicate the way gmail does it since it doesn't care about whether it was a quoted piece or not, like Zac says, it just seems to care about the diff.
Its actually pretty hard to get this right 100% of the time. Plain text email is "lossy", its entirely possible for you to send
> Here is my long line that is over 74 chars (email line length limit)
Which can get encoded as something like
> Here is my long line that is over 74 chars (email=
line length limit)
And then is decoded as
> Here is my long line that is over 74 chars (email
line length limit)
Making it indistinguishable from an inline-reply.
This is email, so variations are abound. Email usually line-wraps at something like 74 characters, and encoding schemes can differ. Its a real PITA. If you can access the HTML version, you will probably have better luck looking for quote tags and the like. Another idea would be to parse both the plain text and html version to try and determine the boundries.
Additionally, its best to just plan for specific client hacks. They all construct mime messages differently, both in structure and header content.
Edit: I say this with the experience of writing an email processing system as well as seeing several people try to do the -exact- thing you're doing. It always only got "ok" results.
From what I can tell, gmail does not bother about prefixed lines or section headings, except to ignore them. If the text lines appeared earlier in the thread, and then reappear, it is considered to be quoted. Thus, e.g., if you send multiple messages and don't change your signature, the signature is considered to be quoted. If you've already dealt with the '>' prefix, a simple diff should do most of the rest. No need to get fancy.
First thing I think I'd do is strip out all the white space, or reduce white space to 1 between each word, and special characters from both blocks, then look for the old one in the new one.
Here's a mozdev project that may be helpful for others who stumble across this page looking for a Thunderbird solution:
http://quotecollapse.mozdev.org/