summaryrefslogtreecommitdiff
path: root/docs/Samba-HOWTO-Collection/Unicode.xml
blob: 7d6abc659cb3c05cf68a1a8a048f1a44ca563311 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE chapter PUBLIC "-//Samba-Team//DTD DocBook V4.2-Based Variant V1.0//EN" "http://www.samba.org/samba/DTD/samba-doc">
<chapter id="unicode">
<chapterinfo>
	&author.jelmer;
	&author.jht;
	<author>
		<firstname>TAKAHASHI</firstname><surname>Motonobu</surname>
		<affiliation>
		<address><email>monyo@home.monyo.com</email></address>
		</affiliation>
		<contrib>Japanese character support</contrib>
	</author>
	<pubdate>25 March 2003</pubdate>
</chapterinfo>

<title>Unicode/Charsets</title>

<sect1>
<title>Features and Benefits</title>

<para>
Every industry eventually matures. One of the great areas of maturation is in
the focus that has been given over the past decade to make it possible for anyone
anywhere to use a computer. It has not always been that way, in fact, not so long
ago it was common for software to be written for exclusive use in the country of
origin.
</para>

<para>
Of all the effort that has been brought to bear on providing native
language support for all computer users, the efforts of the
<ulink url="http://www.openi18n.org/">Openi18n organization</ulink>
is deserving of special mention.
</para>

<para>
Samba-2.x supported a single locale through a mechanism called 
<emphasis>codepages</emphasis>. Samba-3 is destined to become a truly trans-global
file and printer-sharing platform.
</para>

</sect1>

<sect1>
<title>What Are Charsets and Unicode?</title>

<para>
Computers communicate in numbers. In texts, each number will be 
translated to a corresponding letter. The meaning that will be assigned 
to a certain number depends on the <emphasis>character set (charset)
</emphasis> that is used. 
</para>

<para>
A charset can be seen as a table that is used to translate numbers to 
letters. Not all computers use the same charset (there are charsets 
with German umlauts, Japanese characters, and so on). The American Standard Code
for Information Interchange (ASCII) encoding system has been the normative character
encoding scheme used by computers to date. This employs a charset that contains 
256 characters. Using this mode of encoding each character takes exactly one byte.
</para>

<para>
There are also charsets that support extended characters, but those need at least
twice as much storage space as does ASCII encoding. Such charsets can contain
<command>256 * 256 = 65536</command> characters, which is more than all possible
characters one could think of. They are called multi-byte charsets because they use
more then one byte to store one character. 
</para>

<para>
One standardized multi-byte charset encoding scheme is known as
<ulink url="http://www.unicode.org/">unicode</ulink>.  A big advantage of using a
multi-byte charset is that you only need one. There is no need to make sure two
computers use the same charset when they are communicating.
</para>

<para>Old Windows clients use single-byte charsets, named 
<parameter>codepages</parameter>, by Microsoft. However, there is no support for 
negotiating the charset to be used in the SMB/CIFS protocol. Thus, you 
have to make sure you are using the same charset when talking to an older client.
Newer clients (Windows NT, 200x, XP) talk unicode over the wire.
</para>
</sect1>

<sect1>
<title>Samba and Charsets</title>

<para>
As of Samba-3, Samba can (and will) talk unicode over the wire. Internally, 
Samba knows of three kinds of character sets: 
</para>

<variablelist>
	<varlistentry>
		<term><smbconfoption name="unix charset"/></term>
		<listitem><para>
		This is the charset used internally by your operating system. 
		The default is <constant>UTF-8</constant>, which is fine for most 
		systems, which covers all characters in all languages. The default
		in previous Samba releases was to save filenames in the encoding of the 
		clients, for example cp850 for western european countries.
		</para></listitem>
	</varlistentry>

	<varlistentry>
		<term><smbconfoption name="display charset"/></term>
		<listitem><para>This is the charset Samba will use to print messages
		on your screen. It should generally be the same as the <parameter>unix charset</parameter>.
		</para></listitem>
	</varlistentry>

	<varlistentry>
		<term><smbconfoption name="dos charset"/></term>
		<listitem><para>This is the charset Samba uses when communicating with 
		DOS and Windows 9x/Me clients. It will talk unicode to all newer clients.
		The default depends on the charsets you have installed on your system.
		Run <command>testparm -v | grep <quote>dos charset</quote></command> to see 
		what the default is on your system. 
		</para></listitem>
	</varlistentry>
</variablelist>

</sect1>

<sect1>
<title>Conversion from Old Names</title>

<para>Because previous Samba versions did not do any charset conversion, 
characters in filenames are usually not correct in the UNIX charset but only 
for the local charset used by the DOS/Windows clients.</para>

<para>Bjoern Jacke has written a utility named <ulink url="http://j3e.de/linux/convmv/">convmv</ulink>
that can convert whole directory structures to different charsets with one single command. 
</para>

</sect1>

<sect1>
<title>Japanese Charsets</title>

<para>
Setting up Japanese charsets is quite difficult. This is mainly because:
</para>

<itemizedlist>
	<listitem><para>The Windows character set is extended from the original legacy Japanese
		standard (JIS X 0208) and is not standardized. This means that the strictly
		standardized implementation cannot support the full Windows character set.
	</para></listitem>

	<listitem><para> Mainly for historical reasons, there are several encoding methods in
		Japanese, which are not fully compatible with each other. There are
		two major encoding methods. One is the Shift_JIS series, it is used in Windows
		and some UNIX's. The other is the EUC-JP series, used in most UNIX's
		and Linux. Moreover, Samba previously also offered several unique encoding
		methods, named CAP and HEX, to keep interoperability with CAP/NetAtalk and
		UNIX's which can't use Japanese filenames.  Some implementations of the
		EUC-JP series can't support the full Windows character set.
	</para></listitem>

	<listitem><para>There are some code conversion tables between Unicode and legacy
		Japanese character sets. One is compatible with Windows, another one
		is based on the reference of the Unicode consortium and others are 
		a mixed implementation. The Unicode consortium does not officially
		define any conversion tables between Unicode and legacy character
		sets so there cannot be standard one.
	</para></listitem>

	<listitem><para>The character set and conversion tables available in iconv() depends
		on the iconv library that is available. Next to that, the Japanese locale 
		names may be different on different systems.  This means that the value of 
		the charset parameters depends on the implementation of iconv() you are using.
		</para>

		<para>Though 2 byte fixed UCS-2 encoding is used in Windows internally,
		Shift_JIS series encoding is usually used in Japanese environments
		as ASCII encoding is in English environments.
	</para></listitem>
</itemizedlist>

<sect2><title>Basic Parameter Setting</title>

	<para>
	<smbconfoption name="dos charset"/> and 
	<smbconfoption name="display charset"/>
	should be set to the locale compatible with the character set 
	and encoding method used on Windows. This is usually CP932
	but sometimes has a different name.
	</para>

	<para>
	<smbconfoption name="unix charset"/> can be either Shift_JIS series,
	EUC-JP series and UTF-8. UTF-8 is always available but the availability of other locales
	and its name itself depends on the system.
	</para>

	<para>
	Additionally, you can consider to use the Shift_JIS series as the
	value of the <smbconfoption name="unix charset"/>
	parameter by using the vfs_cap module, which does the same thing as
	setting <quote>coding system = CAP</quote> in the Samba 2.2 series.
	</para>

	<para>
	Where to set <smbconfoption name="unix charset"/>
	to is a difficult question. Here is a list of details, advantages and
	disadvantages of using a certain value.
	</para>

	<variablelist>
		<varlistentry><term>Shift_JIS series</term>
			<listitem><para>
			Shift_JIS series means a locale which is equivalent to <constant>Shift_JIS</constant>,
			used as a standard on Japanese Windows. In the case of <constant>Shift_JIS</constant>,
			for example if a Japanese file name consist of 0x8ba4 and 0x974c
			(a 4 bytes Japanese character string meaning <quote>share</quote>) and <quote>.txt</quote>
			is written from Windows on Samba, the file name on UNIX becomes
			0x8ba4, 0x974c, <quote>.txt</quote> (a 8 bytes BINARY string), same as Windows.
			</para>

			<para>Since Shift_JIS series is usually used on some commercial based
			UNIX's; hp-ux and AIX as Japanese locale (however, it is also possible
			to use the EUC-JP series), To use Shift_JIS series on these platforms,
			Japanese file names created from Windows can be referred to also on
			UNIX.</para>

			<para>
			If your UNIX is already working with Shift_JIS and there is a user 
			who needs to use Japanese file names written from Windows, the
			Shift_JIS series is the best choice.  However, broken file names
			may be displayed and some commands which cannot handle non-ASCII
			filenames may be aborted during parsing filenames. especially there
			may be <quote>\ (0x5c)</quote> in file names, which need to be handled carefully.
			So you had better not touch file names written from Windows on UNIX.
			</para>

			<para>
			Note that most Japanized free software actually works with EUC-JP
			only. You had better verify if the Japanized free software can work
			with Shift_JIS.
			</para>
			</listitem>
		</varlistentry>

		<varlistentry><term>EUC-JP series</term>
			<listitem><para>
			EUC-JP series means a locale which is equivalent to the industry
			standard called EUC-JP, widely used in Japanese UNIX (although EUC
			contains specifications for languages other than Japanese, such as
			EUC-KR). In the case of EUC-JP series, for example if a Japanese
			file name consist of 0x8ba4 and 0x974c and <quote>.txt</quote> is written from
			Windows on Samba, the file name on UNIX becomes 0xb6a6, 0xcdad,
			<quote>.txt</quote> (a 8 bytes BINARY string). 
			</para>

			<para>
			Since EUC-JP is usually used on Open source UNIX, Linux and FreeBSD,
			and on commercial based UNIX, Solaris, IRIX and Tru64 UNIX as
			Japanese locale (however, it is also possible on Solaris to use
			Shift_JIS and UTF-8, on Tru64 UNIX to use Shift_JIS). To use EUC-JP
			series, most Japanese file names created from Windows can be
			referred to also on UNIX. Also, most Japanized free software work
			mainly with EUC-JP only. 
			</para>

			<para>
			It is recommended to choose EUC-JP series when using Japanese file
			names on these UNIX.
			</para>

			<para>
			Although there is no character which needs to be carefully treated
			like <quote>\ (0x5c)</quote>, broken file names may be displayed and some
			commands which cannot handle non-ASCII filenames may be aborted
			during parsing filenames.
			</para>

			<para>
			Moreover, if you built Samba using differently installed libiconv,
			eucJP-ms locale included in libiconv and EUC-JP series locale
			included in OS may not be compatible. In this case, you may need to
			avoid using incompatible characters for file names.
			</para>
			</listitem>
		</varlistentry>

		<varlistentry><term>UTF-8</term>
			<listitem><para>
			UTF-8 means a locale which is equivalent to UTF-8, the international
			standard defined by Unicode consortium. In UTF-8, a <parameter>character</parameter> is
			expressed using 1-3 bytes. In case of Japanese, most characters
			are expressed using 3 bytes. Since on Windows Shift_JIS, where a
			character is expressed with 1 or 2 bytes, is used to express
			Japanese, basically a byte length of a UTF-8 string grows 1.5 times
			the length of a original Shift_JIS string. In the case of UTF-8,
			for example if a Japanese file name consist of 0x8ba4 and 0x974c and
			<quote>.txt</quote> is written from Windows on Samba, the file name on UNIX
			becomes 0xe585, 0xb1e6, 0x9c89, <quote>.txt</quote> (a 10 bytes BINARY string).
			</para>

			<para>
			For systems where iconv() is not available or where iconv()'s locales
			are not compatible with Windows, UTF-8 is the only locale available.
			</para>

			<para> 
			There are no systems that use UTF-8 as default locale for Japanese.
			</para>

			<para>
			Some broken file names may be displayed and some commands which
			cannot handle non-ASCII filenames may be aborted during parsing
			filenames. especially there may be <quote>\ (0x5c)</quote> in file names, which
			need to be handled carefully. So you had better not touch file names
			written from Windows on UNIX.
			</para>

			<para>
			In addition, although it is not directly concerned with Samba, since
			there is a delicate difference between iconv() function, which is
			generally used on UNIX and the functions used on other platforms,
			such as Windows and Java about the conversion table between
			Shift_JIS and Unicode, you should be carefully to handle UTF-8.
			</para>

			<para>
			Although Mac OS X uses UTF-8 as its encoding method for filenames,
			it uses an extended UTF-8 specification that Samba cannot handle so
			UTF-8 locale is not available for Mac OS X.
			</para>
			</listitem>
		</varlistentry>

		<varlistentry><term>Shift_JIS series + vfs_cap (CAP encoding)</term>
			<listitem><para>
			CAP encoding means a specification using in CAP and NetAtalk, file
			server software for Macintosh. In the case of CAP encoding, for
			example if a Japanese file name consist of 0x8ba4 and 0x974c and
			<quote>.txt</quote> is written from Windows on Samba, the file name on UNIX
			becomes <quote>:8b:a4:97L.txt</quote> (a 14 bytes ASCII string). 
			</para>

			<para>
			For CAP encoding a byte which cannot be expressed as an ASCII
			character (0x80 or above) is encoded as <quote>:xx</quote> form. You need to take
			care of containing a <quote>\(0x5c)</quote> in a filename but filenames are not
			broken in a system which cannot handle non-ASCII filenames.
			</para>

			<para>
			The greatest merit of CAP encoding is the compatibility of encoding
			filenames with CAP or NetAtalk, file server software of Macintosh.
			Since they usually write a file name on UNIX with CAP encoding, if a
			directory is shared with both Samba and NetAtalk, you need to use
			CAP encoding to avoid non-ASCII filenames are broken.
			</para>

			<para>
			However, recently there are some systems where NetAtalk has been
			patched to write filenames with EUC-JP (i.e. Japanese original Vine Linux).
			Here you need to choose EUC-JP series instead of CAP encoding.
			</para>

			<para>
			vfs_cap itself is available for non Shift_JIS series locales for
			systems which cannot handle non-ASCII characters or systems which
			shares files with NetAtalk.
			</para>

			<para>
			To use CAP encoding on Samba-3, you should use the unix charset parameter and VFS 
			as follows:
			</para>

<smbconfexample><title>VFS CAP</title>
<smbconfsection>[global]</smbconfsection>
<smbconfoption name="dos charset">CP932<footnote><para>the locale name "CP932" may be different name</para></footnote></smbconfoption>
<smbconfoption name="unix charset">CP932</smbconfoption>

<member><para>...</para></member>

<smbconfsection>[cap-share]</smbconfsection>
<smbconfoption name="vfs option">cap</smbconfoption>
</smbconfexample>

			<para>
			You should set CP932 if using GNU libiconv for unix charset. Setting this,
			filenames in the <quote>cap-share</quote> share are written with CAP encoding.
			</para>
			</listitem>
		</varlistentry>
	</variablelist>

</sect2>

<sect2><title>Individual Implementations</title>

<para>
Here is some additional information regarding individual implementations:
</para>

	<variablelist>
		<varlistentry><term>GNU libiconv</term>
			<listitem><para>
			To handle Japanese correctly, you should apply the patch
			<ulink url="http://www2d.biglobe.ne.jp/~msyk/software/libiconv-patch.html">libiconv-1.8-cp932-patch.diff.gz</ulink>
			to libiconv-1.8.
			</para>
			
			<para>
			Using the patched libiconv-1.8, these settings are available:
			</para>


<!-- FIXME: Convert to diagram ? -->
<programlisting>
dos charset = CP932
unix charset = CP932 / eucJP-ms / UTF-8
		|       |
		|       +-- EUC-JP series
		+-- Shift_JIS series
display charset = CP932
</programlisting>

			<para>
			Other Japanese locales (for example Shift_JIS and EUC-JP) should not
			be used for the lack of the compatibility with Windows.
			</para>
			</listitem>
		</varlistentry>

		<varlistentry><term>GNU glibc</term>
			<listitem><para>
			To handle Japanese correctly, you should apply a <ulink url="http://www2d.biglobe.ne.jp/~msyk/software/glibc/">patch</ulink>
			to glibc-2.2.5/2.3.1/2.3.2 or should use the patch-merged versions, glibc-2.3.3 or later.
			</para>

			<para>
			Using the above glibc, these setting are available:
			</para>

<smbconfblock>
<smbconfoption name="dos charset">CP932</smbconfoption>
<smbconfoption name="unix charset">CP932 / eucJP-ms / UTF-8</smbconfoption>
<smbconfoption name="display charset">CP932</smbconfoption>
</smbconfblock>

			<para>
			Other Japanese locales (for example Shift_JIS and EUC-JP) should not
			be used for the lack of the compatibility with Windows.
			</para>
			</listitem>
		</varlistentry>
	</variablelist>

</sect2>

<sect2>
	<title>Migration from Samba-2.2 Series</title>

<para> 
Prior to Samba-2.2 series <quote>coding system</quote> parameter is used as
<smbconfoption name="unix charset"/> parameter of the Samba-3 series.
<link linkend="japancharsets">Next table</link> shows the mapping table when migrating from the Samba-2.2 series to Samba-3.
</para>

	<table frame="all" id="japancharsets">
		<title>Japanese Character Sets in Samba-2.2 and Samba-3</title>

		<tgroup cols="2" align="center">
			<colspec align="center"/>
			<colspec align="center"/>
			<thead>
				<row><entry>Samba-2.2 Coding System</entry><entry>Samba-3 unix charset</entry></row>
			</thead>
			<tbody>
				<row><entry>SJIS</entry><entry>Shift_JIS series</entry></row>
				<row><entry>EUC</entry><entry>EUC-JP series</entry></row>
				<row><entry>EUC3<footnote><para>Only exists in Japanese Samba version</para></footnote></entry><entry>EUC-JP series</entry></row>
				<row><entry>CAP</entry><entry>Shift_JIS series + VFS</entry></row>
				<row><entry>HEX</entry><entry>currently none</entry></row>
				<row><entry>UTF8</entry><entry>UTF-8</entry></row>
				<row><entry>UTF8-Mac<footnote><para>Only exists in Japanese Samba version</para></footnote></entry><entry>currently none</entry></row>
				<row><entry>others</entry><entry>none</entry></row>
			</tbody>
		</tgroup>
	</table>

</sect2>

</sect1>

<sect1>
	<title>Common Errors</title>

	<sect2>
		<title>CP850.so Can't Be Found</title>

		<para><quote>Samba is complaining about a missing <filename>CP850.so</filename> file.</quote></para>

		<para><emphasis>Answer:</emphasis> CP850 is the default <smbconfoption name="dos charset"/>.
		The <smbconfoption name="dos charset"/> is used to convert data to the codepage used by your dos clients.
		If you do not have any dos clients, you can safely ignore this message. </para>

		<para>CP850 should be supported by your local iconv implementation. Make sure you have all the required packages installed.
		If you compiled Samba from source, make sure to configure found iconv.</para>
	</sect2>
</sect1>

</chapter>