From d4b35b895cdf157e49609b59ec89ab648dafb524 Mon Sep 17 00:00:00 2001 From: John Terpstra Date: Wed, 13 Apr 2005 04:04:36 +0000 Subject: More updates. (This used to be commit 20f8bde1d0a2b2e42efedcdac21778fe34c0ab79) --- docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml | 71 +++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml (limited to 'docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml') diff --git a/docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml b/docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml new file mode 100644 index 0000000000..44f054236e --- /dev/null +++ b/docs/Samba-HOWTO-Collection/TOSHARG-LargeFile.xml @@ -0,0 +1,71 @@ + + + + + &author.jeremy; + &author.jht; + March 5, 2005 + +Handling Large Directories + + +Samba-3.0.12 implements a solution for sites that have experienced performance degradation do to the +problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory. + + + +The key was fixing the directory handling to read only the current list requested instead of the old +(up to samba-3.0.11) behaviour of reading the entire directory into memory before doling out names. +Normally this would have broken OS/2 applications which have very strange delete semantics, but by +stealing logic from Samba4 (thanks tridge) the current code in 3.0.12 handles this correctly. + + + +To set up an application that needs large number of files per directory in a way that does not +damage performance unduly follow these steps: + + + +Firstly, you need to canonicalize all the files in the directory to have one case, upper or lower - take your +pick (I chose upper as all my files were already upper case names). Then set up a new custom share for the +application as follows: + +[bigshare] + path = /home/jeremy/tmp/manyfilesdir + read only = no + case sensitive = True + default case = upper + preserve case = no + short preserve case = no + + + + +Of course, use your own path and settings, but set the case options to match the case of all the files in your +directory. The path should point at the large directory needed for the application - any new files created in +there and in any paths under it will be forced by smbd into upper case - but smbd will no longer have to scan +the directory for names - it knows that if a file does not exist in upper case then it doesn't exist at all. + + + +The secret to this is really in the True +line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file +called FOO, and it can not be found by a simple stat call, then smbd will return file not +found immediately without scanning the containing directory for a version of a different case. The other +xxx case xxx lines make this work by forcing a consistent case on all files created by smbd. + + + +Remember, all files and directories under the path directory must be in upper case +with this &smb.conf; stanza as smbd will not be able to find lower case filenames with these settings. Also +note this is done on a per-share basis, allowing this to be set only for a share servicing an application with +this problematic behaviour (using large numbers of entries in a directory) - the rest of your smbd shares +don't need to be affected. + + + +This makes smbd much faster when dealing with large directories. My test case has over 100,000 files and +smbd now deals with this very efficiently. + + + -- cgit