Age | Commit message (Collapse) | Author | Files | Lines |
|
Signed-off-by: Michael Adam <obnox@samba.org>
|
|
Signed-off-by: Michael Adam <obnox@samba.org>
|
|
Signed-off-by: Michael Adam <obnox@samba.org>
|
|
"p" is overwritten further down again before it's first use
Signed-off-by: Michael Adam <obnox@samba.org>
|
|
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Jun 20 10:14:37 CEST 2012 on sn-devel-104
|
|
|
|
|
|
errors
|
|
This just leaves a default enough for the test code to still check the start
of the provision. This may well be removed in future, and we wish to reduce
the extra options to provision.
Andrew Bartlett
|
|
This was part of the now-abandoned S4 AD LDAP backend project.
Andrew Bartlett
|
|
This reverts commit 06c90cb6f55701effa4cbafaf189a4de8471949b.
There is genuine interest in using this currently unused code, so put
it back into the tree to avoid folks having to rewrite it.
It should be carefully hooked back into libnet at some point, and
possibly told how to talk to the s3 nmbd socket if nbt_server isn't
running.
The wscript patches are skipped, due to the way the extra
dep interacted with the build system. When used, this will be resolved.
Andrew Bartlett
|
|
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Jun 19 21:17:51 CEST 2012 on sn-devel-104
|
|
|
|
|
|
|
|
|
|
|
|
Signed-off-by: Jeremy Allison <jra@samba.org>
|
|
Signed-off-by: Jeremy Allison <jra@samba.org>
|
|
|
|
|
|
|
|
|
|
|
|
vlendec@host:~/git/s3> make etags
WAF_MAKE=1 ./buildtools/bin/waf etags
Running: etags $(find /home/vlendec/git/s3 -name "*.[ch]" | egrep -v \.inst\.)
sh: /usr/bin/etags: Argument list too long
'etags' finished successfully (0.368s)
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Jun 19 18:56:53 CEST 2012 on sn-devel-104
|
|
[1427/1518 in 1h12m39s] samba4.nss.test using winbind(member)
UNEXPECTED(failure): samba4.nss.test using winbind(member).run nsstest(member)
REASON: _StringException: _StringException: ERROR setpwent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR getpwent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR endpwent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR setgrent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR getgrent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR endgrent: NSS_STATUS=-1 1 (nss_errno=0)
ERROR Non existent user gave error -1
ERROR Non existent uid gave error -1
ERROR Non existent group gave error -1
ERROR Non existent gid gave error -1
total_errors=10
This seems to happen in all environments
metze
|
|
This reverts commit 846a697e20478798288afb43cdb7a9f389a15c69.
This is still used in source3/smbd/oplock_irix.c
metze
|
|
metze
|
|
This is important in some situations where these queries might be
costly LDAP queries and is just not required for the system token.
This is because the system token should be just the NT
AUTHORITY\SYSTEM user and just enough unix info to allow the token to
be used.
Andrew Bartlett
Signed-off-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Tue Jun 19 13:34:28 CEST 2012 on sn-devel-104
|
|
This is important in some situations where these queries might be
costly LDAP queries and is just not required for the system token.
This is because the system token should be just the NT
AUTHORITY\SYSTEM user and just enough unix info to allow the token to
be used.
We query only NSS to get the name of sec_initial_uid()
Signed-off-by: Andreas Schneider <asn@samba.org>
|
|
With the split up to handle system specially, there is no need for these
static helper functions any more.
Andrew Bartlett
Signed-off-by: Andreas Schneider <asn@samba.org>
|
|
This removes the duplication on how to detect that a user is system in Samba
now that the smbd system account is also only SID_NT_SYSTEM we can use the same
check everywhere.
Andrew Bartlett
Signed-off-by: Andreas Schneider <asn@samba.org>
|
|
This removes the SID matching the sec_initial_uid().
Andrew Bartlett
Signed-off-by: Andreas Schneider <asn@samba.org>
|
|
Signed-off-by: Andreas Schneider <asn@samba.org>
|
|
This is copied from tdb; we build the utilities, but as nothing else
links against it, we shouldn't be adding anything to the normal samba
binary sizes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au>
Autobuild-Date(master): Tue Jun 19 07:31:06 CEST 2012 on sn-devel-104
|
|
This doesn't do anything with it yet, just wires it back into the build.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Simple change, as we get rid of tdb_compat in favour of either ntdb directly
or dbwrap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Simple change, as we get rid of tdb_compat in favour of tdb directly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Update the design.lyx file with the latest status and the change in hashing.
Also, refresh and add examples to the TDB_porting.txt file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
We access the key on lookup, then access the data in the caller. It
makes more sense to access both at once. We also put in a likely()
for the case where the hash is not chained.
Before:
Adding 1000 records: 3644-3724(3675) ns (129656 bytes)
Finding 1000 records: 1596-1696(1622) ns (129656 bytes)
Missing 1000 records: 1409-1525(1452) ns (129656 bytes)
Traversing 1000 records: 1636-1747(1668) ns (129656 bytes)
Deleting 1000 records: 3138-3223(3175) ns (129656 bytes)
Re-adding 1000 records: 3278-3414(3329) ns (129656 bytes)
Appending 1000 records: 5396-5529(5426) ns (253312 bytes)
Churning 1000 records: 9451-10095(9584) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 183881-191112(188223) ops/sec
After:
Adding 1000 records: 3590-3701(3640) ns (129656 bytes)
Finding 1000 records: 1539-1605(1566) ns (129656 bytes)
Missing 1000 records: 1398-1440(1413) ns (129656 bytes)
Traversing 1000 records: 1629-2015(1710) ns (129656 bytes)
Deleting 1000 records: 3118-3236(3163) ns (129656 bytes)
Re-adding 1000 records: 3235-3355(3275) ns (129656 bytes)
Appending 1000 records: 5335-5444(5385) ns (253312 bytes)
Churning 1000 records: 9350-9955(9494) ns (253312 bytes)
smbtorture results (--entries=1000)
ntdb speed 180559-199981(195106) ops/sec
|
|
Since our default hashsize is 8192 not 131, we look fat when we convert
near-empty TDBs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Just like tdbtorture, having a hashsize of 2 stresses us much more!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
Since we've given up on expansion, let them frob the hashsize again.
We have attributes, so we should use them for optional stuff like
this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
|
TDB2 started with a top-level hash of 1024 entries, divided into 128
groups of 8 buckets. When a bucket filled, the 8 bucket group
expanded into pointers into 8 new 64-entry hash tables. When these
filled, they expanded in turn, etc.
It's a nice idea to automatically expand the hash tables, but it
doesn't pay off. Remove it for NTDB.
1) It only beats TDB performance when the database is huge and the
TDB hashsize is small. We are about 20% slower on medium-size
databases (1000 to 10000 records), worse on really small ones.
2) Since we're 64 bits, our hash tables are already twice as expensive
as TDB.
3) Since our hash function is good, it means that all groups tend to
fill at the same time, meaning the hash enlarges by a factor of 128
all at once, leading to a very large database at that point.
4) Our efficiency would improve if we enlarged the top level, but
that makes our minimum db size even worse: it's already over 8k,
and jumps to 1M after about 1000 entries!
5) Making the sub group size larger gives a shallower tree, which
performs better, but makes the "hash explosion" problem worse.
6) The code is complicated, having to handle delete and reshuffling
groups of hash buckets, and expansion of buckets.
7) We have to handle the case where all the records somehow end up with
the same hash value, which requires special code to chain records for
that case.
On the other hand, it would be nice if we didn't degrade as badly as
TDB does when the hash chains get long.
This patch removes the hash-growing code, but instead of chaining like
TDB does when a bucket fills, we point the bucket to an array of
record pointers. Since each on-disk NTDB pointer contains some hash
bits from the record (we steal the upper 8 bits of the offset), 99.5%
of the time we don't need to load the record to determine if it
matches. This makes an array of offsets much more cache-friendly than
a linked list.
Here are the times (in ns) for tdb_store of N records, tdb_store of N
records the second time, and a fetch of all N records. I've also
included the final database size and the smbtorture local.[n]tdb_speed
results.
Benchmark details:
1) Compiled with -O2.
2) assert() was disabled in TDB2 and NTDB.
3) The "optimize fetch" patch was applied to NTDB.
10 runs, using tmpfs (otherwise massive swapping as db hits ~30M,
despite plenty of RAM).
Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046
TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774
NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901
Analysis:
1) TDB wins on small databases, beating TDB2 by ~15%, NTDB by ~10%.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1
|
|
We also split off the NTDB_CONVERT case (where the ntdb is of a
different endian) into its own io function.
NTDB speed:
Adding 10000 records: 3894-9951(8553) ns (815528 bytes)
Finding 10000 records: 1644-4294(3580) ns (815528 bytes)
Missing 10000 records: 1497-4018(3303) ns (815528 bytes)
Traversing 10000 records: 1585-4225(3505) ns (815528 bytes)
Deleting 10000 records: 3088-8154(6927) ns (815528 bytes)
Re-adding 10000 records: 3192-8308(7089) ns (815528 bytes)
Appending 10000 records: 5187-13307(11365) ns (1274312 bytes)
Churning 10000 records: 6772-17567(15078) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 1602-2404(2214) ns (815528 bytes)
Finding 10000 records: 456-871(778) ns (815528 bytes)
Missing 10000 records: 393-522(503) ns (815528 bytes)
Traversing 10000 records: 729-1015(945) ns (815528 bytes)
Deleting 10000 records: 1065-1476(1374) ns (815528 bytes)
Re-adding 10000 records: 1397-1930(1819) ns (815528 bytes)
Appending 10000 records: 2927-3351(3184) ns (1274312 bytes)
Churning 10000 records: 3921-4697(4378) ns (1274312 bytes)
smbtorture results:
ntdb speed 86581-191518(175666) ops/sec
Applying patch..increase-top-level.patch
|
|
The simple "is it in range" check can be inline; complex cases can be
handed through to the normal or transaction handler.
NTDB speed:
Adding 10000 records: 4111-9983(9149) ns (815528 bytes)
Finding 10000 records: 1667-4464(3810) ns (815528 bytes)
Missing 10000 records: 1511-3992(3546) ns (815528 bytes)
Traversing 10000 records: 1698-4254(3724) ns (815528 bytes)
Deleting 10000 records: 3608-7998(7358) ns (815528 bytes)
Re-adding 10000 records: 3259-8504(7805) ns (815528 bytes)
Appending 10000 records: 5393-13579(12356) ns (1274312 bytes)
Churning 10000 records: 6966-17813(16136) ns (1274312 bytes)
NTDB speed in transaction:
Adding 10000 records: 916-2230(2004) ns (815528 bytes)
Finding 10000 records: 330-866(770) ns (815528 bytes)
Missing 10000 records: 196-520(471) ns (815528 bytes)
Traversing 10000 records: 356-879(800) ns (815528 bytes)
Deleting 10000 records: 505-1267(1108) ns (815528 bytes)
Re-adding 10000 records: 658-1681(1477) ns (815528 bytes)
Appending 10000 records: 1088-2827(2498) ns (1274312 bytes)
Churning 10000 records: 1636-4267(3785) ns (1274312 bytes)
smbtorture results:
ntdb speed 85588-189430(157110) ops/sec
|
|
This is designed to allow us to make ntdb_context (and NTDB_DATA returned
from ntdb_fetch) a talloc pointer. But it can also be used for any other
alternate allocator.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|