summaryrefslogtreecommitdiff
path: root/lib/tdb/common
AgeCommit message (Collapse)AuthorFilesLines
2011-06-08tdb: enable VALGRIND to remove valgrind noise.Rusty Russell1-35/+0
Andrew Bartlett complained that valgrind needs --partial-loads-ok=yes otherwise the Jenkins hash makes it complain. My benchmarking here revealed that at least with modern gcc (4.5) and CPU (Intel i5 32 bit) there's no measurable performance penalty for the "correct" code, so rip out the optimized one. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User: Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date: Wed Jun 8 11:05:47 CEST 2011 on sn-devel-104
2011-04-19tdb: make sure we skip over recovery area correctly.Rusty Russell3-17/+44
If it's really the recovery area, we can trust the rec_len field, and don't have to go groping for bitpatterns. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User: Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date: Tue Apr 19 14:15:22 CEST 2011 on sn-devel-104
2011-04-18tdb_expand: limit the expansion with huge recordsSimo Sorce1-5/+20
ldb can create huge records when saving indexes. Limit the tdb expansion to avoid consuming a lot of memory for no good reason if the record being saved is huge.
2011-04-18tdb: tdb_repack() only when it's worthwhile.Rusty Russell1-6/+31
tdb_repack() is expensive and consumes memory, so we can spend some effort to see if it's worthwhile. In particular, tdbbackup doesn't need to repack: it started with an empty database! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-04-18tdb: fix transaction recovery area for converted tdbs.Rusty Russell1-2/+4
This is why macros are dangerous; these were converting the pointers, not the things pointed to! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-03-30tdb: Fix Coverity ID 2238: SECURE_CODINGVolker Lendecke1-24/+24
2011-03-27tdb: Fix Coverity ID 2192: NO_EFFECTVolker Lendecke1-1/+1
(ret < 0) can never be true
2011-02-12tdb: Fix a C++ warningVolker Lendecke1-1/+1
Autobuild-User: Volker Lendecke <vlendec@samba.org> Autobuild-Date: Sat Feb 12 19:50:55 CET 2011 on sn-devel-104
2010-12-29tdb: tdb_summary() support.Rusty Russell3-2/+195
Autobuild-User: Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date: Wed Dec 29 10:12:05 CET 2010 on sn-devel-104
2010-11-27tdb:common/open.c - use "discard_const_p" for certain "tdb->name" assignmentsMatthias Dieter Wallnöfer1-2/+2
In order to suppress compiler warnings.
2010-11-12tdb: set tdb->name early, as it's needed for tdb_name()Stefan Metzmacher1-6/+27
tdb_name() might be used within the given log function, which might be called from within tdb_open_ex(). metze Autobuild-User: Stefan Metzmacher <metze@samba.org> Autobuild-Date: Fri Nov 12 11:22:21 UTC 2010 on sn-devel-104
2010-10-21tdb: Set _PUBLIC_ in C file rather than header files (Debian bug 600898)Jelmer Vernooij11-66/+64
Autobuild-User: Jelmer Vernooij <jelmer@samba.org> Autobuild-Date: Thu Oct 21 11:47:22 UTC 2010 on sn-devel-104
2010-09-27tdb: TDB_INCOMPATIBLE_HASH, to allow safe changing of default hash.Rusty Russell3-4/+20
This flag to tdb_open/tdb_open_ex effects creation of a new database: 1) Uses the Jenkins lookup3 hash instead of the old gdbm hash if none is specified, 2) Places a non-zero field in header->rwlocks, so older versions of TDB will refuse to open it. This means that the caller (ie Samba) can set this flag to safely change the hash function. Versions of TDB from this one on will either use the correct hash or refuse to open (if a different hash is specified). Older TDB versions will see the nonzero rwlocks field and refuse to open it under any conditions. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-27tdb: automatically identify Jenkins hash tdbsRusty Russell1-14/+27
If the caller to tdb_open_ex() doesn't specify a hash, and tdb_old_hash doesn't match, try tdb_jenkins_hash. This was Metze's idea: it makes life simpler, especially with the upcoming TDB_INCOMPATIBLE_HASH flag. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-27tdb: add Bob Jenkins lookup3 hash as helper hash.Rusty Russell3-15/+382
This is a better hash than the default: shipping it with tdb makes it easy for callers to use it as the hash by passing it to tdb_open_ex(). This version taken from CCAN and modified, which took it from http://www.burtleburtle.net/bob/c/lookup3.c. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-20lib/tdb: fix c++ build warning in tdb_header_hash().Günther Deschner1-1/+1
Guenther
2010-09-16tdb: added TDB_NO_FSYNC env variableAndrew Tridgell1-0/+4
this might help reduce test times and load on test machines
2010-09-13tdb: put example hashes into header, so we notice incorrect hash_fn.Rusty Russell3-2/+65
This is Stefan Metzmacher <metze@samba.org>'s patch with minor changes: 1) Use the TDB_MAGIC constant so both hashes aren't of strings. 2) Check the hash in tdb_check (paranoia, really). 3) Additional check in the (unlikely!) case where both examples hash to 0. 4) Cosmetic changes to var names and complaint message. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13tdb: fix tdb_check() on other-endian tdbs.Rusty Russell1-1/+1
We must not endian-convert the magic string, just the rest. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13tdb: fix tdb_check() on read-only TDBs to actually work.Rusty Russell1-5/+17
Commit bc1c82ea137 "Fix tdb_check() to work with read-only tdb databases." claimed to do this, but tdb_lockall_read() fails on read-only databases. Also make sure we can still do tdb_check() inside a transaction (weird, but we previously allowed it so don't break the API). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-09-13tdb: make check more robust against recovery failures.Rusty Russell1-5/+36
We can end up with dead areas when we die during transaction commit; tdb_check() fails on such a (valid) database. This is particularly noticable now we no longer truncate on recovery; if the recovery area was at the end of the file we used to remove it that way. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-08-14tdb: workaround starvation problem in locking entire database.Rusty Russell1-17/+69
We saw tdb_lockall() take 71 seconds under heavy load; this is because Linux (at least) doesn't prevent new small locks being obtained while we're waiting for a big log. The workaround is to do divide and conquer using non-blocking chainlocks: if we get down to a single chain we block. Using a simple test program where children did "hold lock for 100ms, sleep for 1 second" the time to do tdb_lockall() dropped signifiantly. There are ln(hashsize) locks taken in the contended case, but that's slow anyway. More analysis is given in my blog at http://rusty.ozlabs.org/?p=120 This may also help transactions, though in that case it's the initial read lock which uses this gradual locking routine; the update-to-write-lock code is separate and still tries to update in one go. Even though ABI doesn't change, minor version bumped so behavior change can be easily detected. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-07-29Fix tdb_check() to work with read-only tdb databases. The function ↵Jeremy Allison1-3/+3
tdb_lockall() uses F_WRLCK internally, which doesn't work on a fd opened with O_RDONLY. Use tdb_lockall_read() instead. Jeremy.
2010-07-01tdb: fix the build on mac os x 10.6.4.Günther Deschner1-0/+4
Guenther
2010-05-11tdb: remove unused variable in tdb_new_database().Günther Deschner1-1/+0
Guenther
2010-05-05tdb: fix short write logic in tdb_new_databaseRusty Russell3-17/+17
Commit 207a213c/24fed55d purported to fix the problem of signals during tdb_new_database (which could cause a spurious short write, hence a failure). However, the code is wrong: newdb+written is not correct. Fix this by introducing a general tdb_write_all() and using it here and in the tracing code. Cc: Stefan Metzmacher <metze@samba.org> Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-04-20tdb: update tdb ABI to use hide_symbols=TrueAndrew Tridgell1-1/+1
We now use -fvisibilty=hidden to hide symbols from outside the tdb shared library. This also moved tdb_transaction_recover() into the tdb_private.h header, as it should never have been a public API. For that reason we are changing the version number. We're only doing a minor version increment as it is extremely unlikely that anyone was actually using tdb_transaction_recover() as its locking requirements were rather unusual. Pair-Programmed-With: Rusty Russell <rusty@samba.org>
2010-03-26tdb: Add a non-blocking version of tdb_transaction_startVolker Lendecke4-7/+22
2010-03-25tdb: Fix indentation in tdb_new_database()Volker Lendecke1-1/+1
2010-03-25Fix some nonempty blank linesVolker Lendecke10-45/+44
2010-02-28tdb: If tdb_parse_record does not find a record, return -1 instead of 0Volker Lendecke1-1/+4
2010-02-24tdb: handle processes dying during transaction commit.Rusty Russell3-0/+86
tdb transactions were designed to be robust against the machine powering off, but interestingly were never designed to handle the case where an administrator kill -9's a process during commit. Because recovery is only done on tdb_open, processes with the tdb already mapped will simply use it despite it being corrupt and needing recovery. The solution to this is to check for recovery every time we grab a data lock: we could have gained the lock because a process just died. This has no measurable cost: here is the time for tdbtorture -s 0 -n 1 -l 10000: Before: 2.75 2.50 2.81 3.19 2.91 2.53 2.72 2.50 2.78 2.77 = Avg 2.75 After: 2.81 2.57 3.42 2.49 3.02 2.49 2.84 2.48 2.80 2.43 = Avg 2.74 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24patch tdb-refactor-tdb_lock-and-tdb_lock_nonblock.patchRusty Russell1-16/+13
2010-02-24tdb: don't truncate tdb on recoveryRusty Russell1-10/+0
The current recovery code truncates the tdb file on recovery. This is fine if recovery is only done on first open, but is a really bad idea as we move to allowing recovery on "live" databases. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: remove lock opsRusty Russell4-40/+22
Now the transaction code uses the standard allrecord lock, that stops us from trying to grab any per-record locks anyway. We don't need to have special noop lock ops for transactions. This is a nice simplification: if you see brlock, you know it's really going to grab a lock. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: rename tdb_release_extra_locks() to tdb_release_transaction_locks()Rusty Russell3-13/+9
tdb_release_extra_locks() is too general: it carefully skips over the transaction lock, even though the only caller then drops it. Change this, and rename it to show it's clearly transaction-specific. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: cleanup: remove ltype argument from _tdb_transaction_cancel.Rusty Russell1-17/+13
Now the transaction allrecord lock is the standard one, and thus is cleaned in tdb_release_extra_locks(), _tdb_transaction_cancel() doesn't need to know what type it is. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: tdb_allrecord_lock/tdb_allrecord_unlock/tdb_allrecord_upgradeRusty Russell3-29/+62
Centralize locking of all chains of the tdb; rename _tdb_lockall to tdb_allrecord_lock and _tdb_unlockall to tdb_allrecord_unlock, and tdb_brlock_upgrade to tdb_allrecord_upgrade. Then we use this in the transaction code. Unfortunately, if the transaction code records that it has grabbed the allrecord lock read-only, write locks will fail, so we treat this upgradable lock as a write lock, and mark it as upgradable using the otherwise-unused offset field. One subtlety: now the transaction code is using the allrecord_lock, the tdb_release_extra_locks() function drops it for us, so we no longer need to do it manually in _tdb_transaction_cancel. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: suppress record write locks when allrecord lock is taken.Rusty Russell1-0/+9
Records themselves get (read) locked by the traversal code against delete. Interestingly, this locking isn't done when the allrecord lock has been taken, though the allrecord lock until recently didn't cover the actual records (it now goes to end of file). The write record lock, grabbed by the delete code, is not suppressed by the allrecord lock. This is now bad: it causes us to punch a hole in the allrecord lock when we release the write record lock. Make this consistent: *no* record locks of any kind when the allrecord lock is taken. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: cleanup: always grab allrecord lock to infinity.Rusty Russell1-7/+3
We were previously inconsistent with our "global" lock: the transaction code grabbed it from FREELIST_TOP to end of file, and the rest of the code grabbed it from FREELIST_TOP to end of the hash chains. Change it to always grab to end of file for simplicity and so we can merge the two. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: remove num_locksRusty Russell2-11/+2
This was redundant before this patch series: it mirrored num_lockrecs exactly. It still does. Also, skip useless branch when locks == 1: unconditional assignment is cheaper anyway. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: use tdb_nest_lock() for seqnum lock.Rusty Russell1-3/+3
This is pure overhead, but it centralizes the locking. Realloc (esp. as most implementations are lazy) is fast compared to the fnctl anyway. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: use tdb_nest_lock() for active lock.Rusty Russell2-5/+18
Use our newly-generic nested lock tracking for the active lock. Note that the tdb_have_extra_locks() and tdb_release_extra_locks() functions have to skip over this lock now it is tracked. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-22tdb: use tdb_nest_lock() for open lock.Rusty Russell3-15/+10
This never nests, so it's overkill, but it centralizes the locking into lock.c and removes the ugly flag in the transaction code to track whether we have the lock or not. Note that we have a temporary hack so this places a real lock, despite the fact that we are in a transaction. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: use tdb_nest_lock() for transaction lock.Rusty Russell2-32/+23
Rather than a boutique lock and a separate nest count, use our newly-generic nested lock tracking for the transaction lock. Note that the tdb_have_extra_locks() and tdb_release_extra_locks() functions have to skip over this lock now it is tracked. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: cleanup: find_nestlock() helper.Rusty Russell1-28/+23
Factor out two loops which find locks; we are going to introduce a couple more so a helper makes sense. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-24tdb: cleanup: tdb_release_extra_locks() helperRusty Russell3-17/+22
Move locking intelligence back into lock.c, rather than open-coding the lock release in transaction.c. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: cleanup: tdb_have_extra_locks() helperRusty Russell4-5/+17
In many places we check whether locks are held: add a helper to do this. The _tdb_lockall() case has already checked for the allrecord lock, so the extra work done by tdb_have_extra_locks() is merely redundant. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: don't suppress the transaction lock because of the allrecord lock.Rusty Russell1-6/+0
tdb_transaction_lock() and tdb_transaction_unlock() do nothing if we hold the allrecord lock. However, the two locks don't overlap, so this is wrong. This simplification makes the transaction lock a straight-forward nested lock. There are two callers for these functions: 1) The transaction code, which already makes sure the allrecord_lock isn't held. 2) The traverse code, which wants to stop transactions whether it has the allrecord lock or not. There have been deadlocks here before, however this should not bring them back (I hope!) Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2010-02-17tdb: cleanup: tdb_nest_lock/tdb_nest_unlockRusty Russell3-45/+67
Because fcntl locks don't nest, we track them in the tdb->lockrecs array and only place/release them when the count goes to 1/0. We only do this for record locks, so we simply place the list number (or -1 for the free list) in the structure. To generalize this: 1) Put the offset rather than list number in struct tdb_lock_type. 2) Rename _tdb_lock() to tdb_nest_lock, make it non-static and move the allrecord check out to the callers (except the mark case which doesn't care). 3) Rename _tdb_unlock() to tdb_nest_unlock(), make it non-static and move the allrecord out to the callers (except mark again). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
generated by cgit (git 2.34.1) at 2024-11-14 21:45:12 +0000