diff options
Diffstat (limited to 'docs-xml/Samba3-HOWTO/TOSHARG-LargeFile.xml')
-rw-r--r-- | docs-xml/Samba3-HOWTO/TOSHARG-LargeFile.xml | 89 |
1 files changed, 89 insertions, 0 deletions
diff --git a/docs-xml/Samba3-HOWTO/TOSHARG-LargeFile.xml b/docs-xml/Samba3-HOWTO/TOSHARG-LargeFile.xml new file mode 100644 index 0000000000..5fdbe7a243 --- /dev/null +++ b/docs-xml/Samba3-HOWTO/TOSHARG-LargeFile.xml @@ -0,0 +1,89 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE chapter PUBLIC "-//Samba-Team//DTD DocBook V4.2-Based Variant V1.0//EN" "http://www.samba.org/samba/DTD/samba-doc"> +<chapter id="largefile"> +<chapterinfo> + &author.jeremy; + &author.jht; + <pubdate>March 5, 2005</pubdate> +</chapterinfo> +<title>Handling Large Directories</title> + +<para> +<indexterm><primary>performance degradation</primary></indexterm> +<indexterm><primary>large numbers of files</primary></indexterm> +<indexterm><primary>large directory</primary></indexterm> +Samba-3.0.12 and later implements a solution for sites that have experienced performance degradation due to the +problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory. +</para> + +<para> +<indexterm><primary>read directory into memory</primary></indexterm> +<indexterm><primary>strange delete semantics</primary></indexterm> +The key was fixing the directory handling to read only the current list requested instead of the old +(up to samba-3.0.11) behavior of reading the entire directory into memory before doling out names. +Normally this would have broken OS/2 applications, which have very strange delete semantics, but by +stealing logic from Samba4 (thanks, Tridge), the current code in 3.0.12 handles this correctly. +</para> + +<para> +<indexterm><primary>large directory</primary></indexterm> +<indexterm><primary>performance</primary></indexterm> +To set up an application that needs large numbers of files per directory in a way that does not +damage performance unduly, follow these steps: +</para> + +<para> +<indexterm><primary>canonicalize files</primary></indexterm> +First, you need to canonicalize all the files in the directory to have one case, upper or lower &smbmdash; take your +pick (I chose upper because all my files were already uppercase names). Then set up a new custom share for the +application as follows: +<smbconfblock> +<smbconfsection name="[bigshare]"/> +<smbconfoption name="path">/data/manyfilesdir</smbconfoption> +<smbconfoption name="read only">no</smbconfoption> +<smbconfoption name="case sensitive">True</smbconfoption> +<smbconfoption name="default case">upper</smbconfoption> +<smbconfoption name="preserve case">no</smbconfoption> +<smbconfoption name="short preserve case">no</smbconfoption> +</smbconfblock> +</para> + +<para> +<indexterm><primary>case options</primary></indexterm> +<indexterm><primary>match case</primary></indexterm> +<indexterm><primary>uppercase</primary></indexterm> +Of course, use your own path and settings, but set the case options to match the case of all the files in your +directory. The path should point at the large directory needed for the application &smbmdash; any new files created in +there and in any paths under it will be forced by smbd into uppercase, but smbd will no longer have to scan +the directory for names: it knows that if a file does not exist in uppercase, then it doesn't exist at all. +</para> + +<para> +<indexterm><primary>case-insensitive</primary></indexterm> +<indexterm><primary>consistent case</primary></indexterm> +<indexterm><primary>smbd</primary></indexterm> +The secret to this is really in the <smbconfoption name="case sensitive">True</smbconfoption> +line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file +called <filename>FOO</filename>, and it cannot be found by a simple stat call, then smbd will return file not +found immediately without scanning the containing directory for a version of a different case. The other +<filename>xxx case xxx</filename> lines make this work by forcing a consistent case on all files created by +&smbd;. +</para> + +<para> +<indexterm><primary>uppercase</primary></indexterm> +<indexterm><primary>stanza</primary></indexterm> +<indexterm><primary>lowercase filenames</primary></indexterm> +Remember, all files and directories under the <parameter>path</parameter> directory must be in uppercase +with this &smb.conf; stanza because &smbd; will not be able to find lowercase filenames with these settings. Also +note that this is done on a per-share basis, allowing this parameter to be set only for a share servicing an application with +this problematic behavior (using large numbers of entries in a directory) &smbmdash; the rest of your &smbd; shares +don't need to be affected. +</para> + +<para> +This makes smbd much faster when dealing with large directories. My test case has over 100,000 files, and +smbd now deals with this very efficiently. +</para> + +</chapter> |