How to remove CTRL-M (^M) characters from a file in Linux

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12
Novell GroupWise 2012
Novell GroupWise 8
Novell Data Synchronizer Mobility Pack
Novell GroupWise Mobility Service

Situation

How to remove CTRL-M (^M) blue carriage return characters from a file in Linux.
How to convert plain text files in DOS/MAC format to UNIX format.
Newly created SSL Certificates not working properly.
Certs invalid or not properly configured, agents unable to use.
Application reporting obscure syntax errors and other unanticipated, unforeseen errors.
Viewing the certificate files in Linux shows ^M characters appended to every line.
The file in question was created in Windows and then copied over to Linux.
^M is the keyboard equivalent to \r or CTRL-v + CTRL-m in vim.

Resolution

There are several ways to translate a file between ASCII CR+LF (DOS/Windows) and LF (Unix) newlines. The following are different options to remove, convert or translate the ^M characters:

  • The simplest solution, use the dos2unix command (sometimes named fromdos, d2u or unix2dos):
    dos2unix filename
  • Using the stream editor sed:
    sed -e "s/\r//g" file > newfile

  • Using perl:
    perl -p -e 's/\r//g' file > newfile
  • Using a terminal editor vi or vim:
    Inside vi [in ESC mode] type: :%s/^M//g
    Note: To enter ^M, type CTRL-V + M. That is, hold down the CTRL key then press V and M in succession.

Cause

File has been transferred between systems of different types with different newline conventions. For example, Windows-based text editors will have a special carriage return character (CR+LF) at the end of lines to denote a line return or newline, which will be displayed incorrectly in Linux (^M). This can be difficult to spot, as some applications or programs may handle the foreign newline characters properly while others do not. Thus some services may crash or not respond correctly. Often times, this is because the file is created or perhaps even edited on a Microsoft Windows machine and then uploaded or transferred to a Linux server. This typically occurs when a file is transferred from MS-DOS (or MS-Windows) without ASCII or text mode.

Additional Information

For debugging purposes on a Unix system, the following command will make the ^M character visible:
cat -v <filename>
 
To enter the ^M character on Linux terminal, press CTRL-V + M at the same time.
 
Here is an example of what these hidden characters look like:

-----BEGIN CERTIFICATE-----^M
MIIDBjCCAe4CCQD3cWtEoYMtujANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB^M
VTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0^M
cyBQdHkgTHRkMB4XDTEzMDcxNzIxMTU0NFoXDTE1MDcxNzIxMTU0NFowRTELMAkG^M
A1UEBhMCQVUxEzARBgNVBAgTClNvbWUtU3RhdGUxITAfBgNVBAoTGEludGVybmV0^M
IFdpZGdpdHMgUHR5IEx0ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB^M
ANIQ4wo6zi9lY5WddpJbHO2OvXgbJZIB8qidksKrFm3tUMe6LiXpOb4fZAwqlUUp^M
fN+jrp+Lr7PPF52vnbKMP4dq31CWV68EtcRK3O8maLy4mpx7pCXeZIqRw3/1UwIr^M
RXzOCWTsdvudAC0Sp0DgN8U7qECojZso9nu1SBopOtBDDU/Wcdmd3ydZJN0R7waB^M
e38WNK6rUNCgcIP0obvuiwppJmUyE1hr09A3W2zGB6bMF4EqZ5uaiBoPhMyKTlDs^M
UMOd6tRXkrYEdnLuqQttUvtE6sfxZaUTmod5MCXSRrW0tAkBGhqlPBT8NvlRQzaz^M
TqzLqhCYPQ9HBihumKHRkucCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEALAGKt8Bn^M
PHYPj6RNioE8eyCvx4WFG8zDIkbkFbmpBiq6SOpWliCnhGWyH5yFEDWHCOebz8L9^M
LHQTi18KDGk7PH0BpWZDpksu3JrBb3fjm+1QCpD9KWRYJVnBGAaWFFmPPpJJDoz2^M
X0MRPbzxfqsno68l46X+5l5m4H8Y2g0cOUto580x/ZlNb+gHbTzuBnwGa2Oof0bc^M
A7ZmU1IytADXWVM28h88XcBooT4dwxCVkwznNGQ8lQ+XTehjdcDEEvbTB/Jc+1M4^M
0a/OvmlDIFU34DKH/LRc7bmTe4YTTRtU5ShXTBH0ChZFuqP20vtiEsPHVn+pk4LN^M
129c5uhrxPGhNQ==^M
-----END CERTIFICATE-----^M

For an in-depth explanation of line endings, see Newline from Wikipedia.
0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
Linux - How to monitor memory usage
Viewed 1293 times since Fri, Jun 8, 2018
high swap space utilization in LINUX
Viewed 4885 times since Fri, Jul 13, 2018
Linux – delete the LUN and remove traces from OS
Viewed 1349 times since Tue, May 22, 2018
Extending Linux LVM partitions - scripts
Viewed 4194 times since Sun, May 20, 2018
LVM: Rename root VG/LV
Viewed 4953 times since Sat, Jun 2, 2018
Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel
Viewed 1117 times since Sun, Dec 6, 2020
Odpalenie polecenia tylko na jedną godzinę
Viewed 1505 times since Thu, May 24, 2018
RHCS6: Create a new Logical Volume / Global Filesystem 2 (GFS2)
Viewed 1122 times since Sun, Jun 3, 2018
OEL 7 – How to disable IPv6 on Oracle Linux 7 – Follow Up
Viewed 7758 times since Wed, Jul 25, 2018
RHCS6: ’fencing’ basics
Viewed 1186 times since Sun, Jun 3, 2018