Ruby Programming/Encoding

Template:Encoding

Encoding SupportEdit

With the advent of Ruby 1.9, Ruby now supports encodings other than US-ASCII for strings, IO, and source code. On an installation of Ruby 1.9.3 MRI on Mac OSX Lion, Ruby has the following encodings:

  • ASCII-8BIT
  • UTF-8
  • US-ASCII
  • Big5
  • Big5-HKSCS
  • Big5-UAO
  • CP949
  • Emacs-Mule
  • EUC-JP
  • EUC-KR
  • EUC-TW
  • GB18030
  • GBK
  • ISO-8859-1
  • ISO-8859-2
  • ISO-8859-3
  • ISO-8859-4
  • ISO-8859-5
  • ISO-8859-6
  • ISO-8859-7
  • ISO-8859-8
  • ISO-8859-9
  • ISO-8859-10
  • ISO-8859-11
  • ISO-8859-13
  • ISO-8859-14
  • ISO-8859-15
  • ISO-8859-16
  • KOI8-R
  • KOI8-U
  • Shift_JIS
  • UTF-16BE
  • UTF-16LE
  • UTF-32BE
  • UTF-32LE
  • Windows-1251
  • IBM437
  • IBM737
  • IBM775
  • CP850
  • IBM852
  • CP852
  • IBM855
  • CP855
  • IBM857
  • IBM860
  • IBM861
  • IBM862
  • IBM863
  • IBM864
  • IBM865
  • IBM866
  • IBM869
  • Windows-1258
  • GB1988
  • macCentEuro
  • macCroatian
  • macCyrillic
  • macGreek
  • macIceland
  • macRoman
  • macRomania
  • macThai
  • macTurkish
  • macUkraine
  • CP950
  • CP951
  • stateless-ISO-2022-JP
  • eucJP-ms
  • CP51932
  • GB2312
  • GB12345
  • ISO-2022-JP
  • ISO-2022-JP-2
  • CP50220
  • CP50221
  • Windows-1252
  • Windows-1250
  • Windows-1256
  • Windows-1253
  • Windows-1255
  • Windows-1254
  • TIS-620
  • Windows-874
  • Windows-1257
  • Windows-31J
  • MacJapanese
  • UTF-7
  • UTF8-MAC
  • UTF-16
  • UTF-32
  • UTF8-DoCoMo
  • SJIS-DoCoMo
  • UTF8-KDDI
  • SJIS-KDDI
  • ISO-2022-JP-KDDI
  • stateless-ISO-2022-JP-KDDI
  • UTF8-SoftBank
  • SJIS-SoftBank

Using EncodingsEdit

By default, all Ruby source files are encoded with US-ASCII. The usual way of changing the encoding of a file is to use a so called "magic comment". The magic comment must come directly at the beginning of the file, or directly after a [comment]. The syntax of the magic comment requires only one thing: The comment contains the text coding: followed by the name of the encoding. So the following are all valid magic comments:

#encoding: UTF-8
#coding: UTF-8
#blah blah coding: US-ASCII

The magic comment tells the interpreter that the source code itself, in addition to all of the strings in it, are going to be encoded with the given encoding. So, while this is valid code:

#encoding: ISO-8891-1
puts "Olé!"

This is not:

#encoding: US-ASCII
puts "Olé!"

Because the first snippet declares the files encoding to be ISO-8891-1 (an extension to US-ASCII that adds accented characters for languages such as French and Spanish), the character "é" is valid. However, in US-ASCII, "é" is an invalid character, and will cause an error.

Encodings and Individual StringsEdit

You can also specify encodings for individual strings in your file (although the characters in literals must still be in the encoding declared by the magic comment, or inserted through escape sequences). This is done by two methods of the String class: encode and force_encoding.

encode is used for transcoding. Given, say, the ISO-8891-1 string "Olé!", you could use encode to convert that to UTF-8, which has all of the same characters. However, you can not transcode the ISO-8891-1 string to US-ASCII, unless it contains only ASCII characters (eg. "Hello"). encode has many options and can be configured extensively. See its documentation [[1]]. Here's the catch of encode: It is very likely that while the visual display and meaning of the characters remain the same, the underlying bytes most likely will not. encode is free to change the underlying bytes of a string. Example:

#encoding: ISO-8891-1
"Olé!".encode("UTF-8") #Valid
"Olé!".encode("US-ASCII") #Error

force_encoding is used to tell Ruby the encoding of a string that already has the correct bytes for that encoding (eg. a UTF-8 string read from a file in an ISO-8891-1 program). force_encoding will never modify the underlying bytes of a string. Example:

#encoding: ISO-8891-1
"\u27d0".force_encoding("UTF-8")

ASCII-8 BitEdit

Ruby also includes a fake encoding: ASCII-8 Bit, or BINARY. BINARY is the encoding used for binary data.

Last modified on 13 August 2012, at 21:32