s3:dbwrap_ctdb: fix race condition with concurrent transactions on the same node.

In ctdb_transaction_commit(), when the trans2_commit control fails, there
is a race condition in the 1 second sleep between the local transaction_cancel
and the call to ctdb_replay_transaction(): The database is not locked, and
neither is the transaction_lock record. So another client can start and possibly
complete a new transaction in this gap, but only on the same node: The locking
of the transaction_lock record on a different node which involves migration of
the record to the other node has been disabled by introduction of the
transaction_active flag on the db which closes precisely this gap from the start
of the commit until the call to TRANS2_FINISH or TRANS2_ERROR.
But this mechanism does not cover the case where a process on the same node
tries to start a transaction: There is no obstacle to locking the transaction_lock
record because the record does not need to be migrated.

This commit closes this race condition in ctdb_transaction_fetch_start()
by using the new ctdb_ctrl_transaction_active() call to ask the local
ctdb daemon whether it has a transaction running on the database.
If so, the check is repeated until the running transaction is done.

This does introduce an additional call to the local ctdbd when starting
transactions, but it does close the (hopefully) last race condition.

Michael

1 files changed, 12 insertions, 0 deletions

diff --git a/source3/lib/dbwrap_ctdb.c b/source3/lib/dbwrap_ctdb.c
index ec52d766c0..fef984c8c3 100644
--- a/source3/lib/dbwrap_ctdb.c
+++ b/source3/lib/dbwrap_ctdb.c
@@ -347,6 +347,7 @@ static int db_ctdb_transaction_fetch_start(struct db_ctdb_transaction_handle *h)
 	pid_t pid;
 	NTSTATUS status;
 	struct ctdb_ltdb_header header;
+	int32_t transaction_status;
 
 	key.dptr = (uint8_t *)discard_const(keyname);
 	key.dsize = strlen(keyname);
@@ -362,6 +363,17 @@ again:
 	}
 	crec = talloc_get_type_abort(rh->private_data, struct db_ctdb_rec);
 
+	transaction_status = db_ctdb_transaction_active(ctx->db_id);
+	if (transaction_status == 1) {
+		unsigned long int usec = (1000 + random()) % 100000;
+		DEBUG(3, ("Transaction already active on db_id[0x%08x]."
+			  "Re-trying after %lu microseconds...",
+			  ctx->db_id, usec));
+		talloc_free(tmp_ctx);
+		usleep(usec);
+		goto again;
+	}
+
 	/*
 	 * store the pid in the database:
 	 * it is not enought that the node is dmaster...