r2716: created a separate detailed talloc_guide.txt document, after volker

complained it was all too confusing :-)

I recommend that everyone who wants to work on Samba4 have a read of this.
(This used to be commit c4c427576c02b27d829ae4aaee31cbf893b4a2ad)

2 files changed, 365 insertions, 57 deletions

diff --git a/prog_guide.txt b/prog_guide.txt
index 4bf8671e60..31a9fb8861 100644
--- a/prog_guide.txt
+++ b/prog_guide.txt
@@ -191,54 +191,8 @@ in the data and bss columns in "size" anyway (it will be included in
 How to use talloc
 -----------------
 
-If you are used to talloc from Samba3 then please read this carefully,
-as talloc has changed rather a lot.
-
-The new talloc is a hierarchical, reference counted memory pool system
-with destructors. Quite a mounthful really, but not too bad once you
-get used to it.
-
-Perhaps the biggest change from Samba3 is that there is no distinction
-between a "talloc context" and a "talloc pointer". Any pointer
-returned from talloc() is itself a valid talloc context. This means
-you can do this:
-
-  struct foo *a = talloc(mem_ctx, sizeof(*s));
-  a->name = talloc(a, strlen("foo")+1);
-
-and the pointer a->name would be a "child" of the talloc context "a"
-which is itself a child of mem_ctx. So if you do talloc_free(mem_ctx)
-then it is all destroyed, whereas if you do talloc_free(a) then just a
-and a->name are destroyed.
-
-If you think about this, then what this effectively gives you is an
-n-ary tree, where you can free any part of the tree with
-talloc_free().
-
-The next big change with the new talloc is reference counts. A talloc
-pointer starts with a reference count of 1. You can call
-talloc_increase_ref_count() on any talloc pointer and that increases
-the reference count by 1. If you then call talloc_free() on a pointer
-that has a reference count greater than 1, then the reference count is
-decreased, but the memory is not released.
-
-Finally, talloc now has destructors. You can set a destructor on any
-talloc pointer using talloc_set_destructor(). Your destructor will
-then be called before the memory is released. An interesting feature
-of these destructors is that they can return a error. If the
-destructor returns -1 then that is interpreted as a refusal to release
-the memory, and the talloc_free() will return. It will also prevent
-the release of all memory "below" that memory in the tree.
-
-You should also go and look at a new talloc function in Samba4 called
-talloc_steal(). By using talloc_steal() you can move a lump of memory
-from one memory context to another without copying the data. This
-should be used when a backend function (such as a packet parser)
-produces a result as a lump of talloc memory and you need to keep it
-around for a longer lifetime than the talloc context it is in. You
-just "steal" the memory from the short-lived context, putting it into
-your long lived context.
-
+Please see the separate document, talloc_guide.txt in this
+directory. You _must_ read this if you want to program in Samba4.
 
 Interface Structures
 --------------------
@@ -583,9 +537,11 @@ other recognised flags are:
 
   sign : enable ntlmssp signing
   seal : enable ntlmssp sealing
+  connect : enable rpc connect level auth (auth, but no sign or seal)
   validate: enable the NDR validator
   print: enable debugging of the packets
   bigendian: use bigendian RPC
+  padcheck: check reply data for non-zero pad bytes
 
 
 For example, these all connect to the samr pipe:
@@ -645,8 +601,6 @@ MSRPC
  - msrpc
 
 
-- use _p talloc varients
-
 don't zero structures! avoid ZERO_STRUCT() and talloc_zero()
 
 
@@ -656,8 +610,6 @@ put in full UNC path in tconx
 
 test timezone handling by using a server in different zone from client
 
-don't just use any old TALLOC_CTX, use the right one!
-
 do {} while (0) system
 
 NT_STATUS_IS_OK() is NOT the opposite of NT_STATUS_IS_ERR()
@@ -665,8 +617,6 @@ NT_STATUS_IS_OK() is NOT the opposite of NT_STATUS_IS_ERR()
 need to implement secondary parts of trans2 and nttrans in server and
 client
 
-add talloc_steal() to move a talloc ptr from one pool to another
-
 document access_mask in openx reply
 
 check all capabilities and flag1, flag2 fields (eg. EAs)
@@ -803,7 +753,6 @@ Ideas
 
 
 BUGS:
-  non-signed non-sealed RPC (level == 2 == "connect")
   add a test case for last_entry_offset in trans2 find interfaces
   conn refused
   connect -> errno
@@ -814,5 +763,3 @@ BUGS:
      trans2 and other calls
   handle servers that don't have the setattre call in torture
   add max file coponent length test and max path len test
-
-
diff --git a/talloc_guide.txt b/talloc_guide.txt
new file mode 100644
index 0000000000..925f485997
--- /dev/null
+++ b/talloc_guide.txt
@@ -0,0 +1,361 @@
+Using talloc in Samba4
+----------------------
+
+Andrew Tridgell
+September 2004
+
+The most current version of this document is available at
+   http://samba.org/ftp/unpacked/samba4/talloc_guide.txt
+
+If you are used to talloc from Samba3 then please read this carefully,
+as talloc has changed a lot.
+
+The new talloc is a hierarchical, reference counted memory pool system
+with destructors. Quite a mounthful really, but not too bad once you
+get used to it.
+
+Perhaps the biggest change from Samba3 is that there is no distinction
+between a "talloc context" and a "talloc pointer". Any pointer
+returned from talloc() is itself a valid talloc context. This means
+you can do this:
+
+  struct foo *X = talloc_p(mem_ctx, struct foo);
+  X->name = talloc_strdup(X, "foo");
+
+and the pointer X->name would be a "child" of the talloc context "X"
+which is itself a child of mem_ctx. So if you do talloc_free(mem_ctx)
+then it is all destroyed, whereas if you do talloc_free(X) then just X
+and X->name are destroyed, and if you do talloc_free(X->name) then
+just the name element of X is destroyed.
+
+If you think about this, then what this effectively gives you is an
+n-ary tree, where you can free any part of the tree with
+talloc_free().
+
+If you find this confusing, then I suggest you run the LOCAL-TALLOC
+smbtorture test with the --leak-report-full option to watch talloc in
+action. You may also like to add your own tests to
+source/torture/local/talloc.c to clarify how some particular situation
+is handled.
+
+
+talloc API
+----------
+
+The following is a complete guide to the talloc API. Read it all at
+least twice.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc(const void *context, size_t size);
+
+The talloc() function is the core of the talloc library. It takes a
+memory context, and returns a pointer to a new area of memory of the
+given size.
+
+The returned pointer is itself a talloc context, so you can use it as
+the context argument to more calls to talloc if you wish.
+
+The returned pointer is a "child" of the supplied context. This means
+that if you talloc_free() the context then the new child disappears as
+well. Alternatively you can free just the child.
+
+The context argument to talloc() can be NULL, in which case a new top
+level context is created. 
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+int talloc_free(void *ptr);
+
+The talloc_free() function frees a piece of talloc memory, and all its
+children. You can call talloc_free() on any pointer returned by
+talloc().
+
+The return value of talloc_free() indicates success or failure, with 0
+returned for success and -1 for failure. The only possible failure
+condition is if the pointer had a destructor attached to it and the
+destructor returned -1. See talloc_set_destructor() for details on
+destructors.
+
+If this pointer has an additional reference when talloc_free() is
+called then the memory is not actually released, but instead the
+reference is destroyed and the memory becomes a child of the
+referrer. See talloc_reference() for details on establishing
+additional references.
+
+talloc_free() operates recursively on its children.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_reference(const void *context, const void *ptr);
+
+The talloc_reference() function returns an additional reference to
+"ptr", and makes this additional reference a child of "context".
+
+The return value of talloc_reference() is always the original pointer
+"ptr", unless talloc ran out of memory in creating the reference in
+which case it will return NULL (each additional reference consumes
+around 48 bytes of memory on intel x86 platforms).
+
+After creating a reference you can free it in one of the following
+ways:
+
+  - you can talloc_free() a parent of the original pointer. That will
+    destroy the reference and make the pointer a child of "context".
+
+  - you can talloc_free() the pointer itself. That will destroy the
+    reference and make the pointer a child of "context".
+
+  - you can talloc_free() the context where you placed the
+    reference. That will destroy the reference, and leave the pointer
+    where it is.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_set_destructor(const void *ptr, int (*destructor)(void *));
+
+The function talloc_set_destructor() sets the "destructor" for the
+pointer "ptr". A destructor is a function that is called when the
+memory used by a pointer is about to be released. The destructor
+receives the pointer as an argument, and should return 0 for success
+and -1 for failure.
+
+The destructor can do anything it wants to, including freeing other
+pieces of memory. A common use for destructors is to clean up
+operating system resources (such as open file descriptors) contained
+in the structure the destructor is placed on.
+
+You can only place one destructor on a pointer. If you need more than
+one destructor then you can create a zero-length child of the pointer
+and place an additional destructor on that.
+
+To remove a destructor call talloc_set_destructor() with NULL for the
+destructor.
+
+If your destructor attempts to talloc_free() the pointer that it is
+the destructor for then talloc_free() will return -1 and the free will
+be ignored. This would be a pointless operation anyway, as the
+destructor is only called when the memory is just about to go away.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_increase_ref_count(const void *ptr);
+
+The talloc_increase_ref_count(ptr) function is exactly equivalent to:
+
+  talloc_reference(NULL, ptr);
+
+You can use either syntax, depending on which you think is clearer in
+your code.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_set_name(const void *ptr, const char *fmt, ...);
+
+Each talloc pointer has a "name". The name is used principally for
+debugging purposes, although it is also possible to set and get the
+name on a pointer in as a way of "marking" pointers in your code.
+
+The main use for names on pointer is for "talloc reports". See
+talloc_report() and talloc_report_full() for details. Also see
+talloc_enable_leak_report() and talloc_enable_leak_report_full().
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_set_name_const(const void *ptr, const char *name);
+
+The function talloc_set_name_const() is just like talloc_set_name(),
+but it takes a string constant, and is much faster. It is extensively
+used by the "auto naming" macros, such as talloc_p().
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_named(const void *context, size_t size, const char *fmt, ...);
+
+The talloc_named() function creates a named talloc pointer. It is
+equivalent to:
+
+   ptr = talloc(context, size);
+   talloc_set_name(ptr, fmt, ....);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_named_const(const void *context, size_t size, const char *name);
+
+This is equivalent to:
+
+   ptr = talloc(context, size);
+   talloc_set_name_const(ptr, name);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+const char *talloc_get_name(const void *ptr);
+
+This returns the current name for the given talloc pointer. See
+talloc_set_name() for details.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_init(const char *fmt, ...);
+
+This function creates a zero length named talloc context as a top
+level context. It is equivalent to:
+
+  talloc_named(NULL, 0, fmt, ...);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_realloc(const void *context, void *ptr, size_t size);
+
+The talloc_realloc() function changes the size of a talloc
+pointer. It has the following equivalences:
+
+  talloc_realloc(context, NULL, size) ==> talloc(context, size);
+  talloc_realloc(context, ptr, 0)     ==> talloc_free(ptr);
+
+The "context" argument is only used if "ptr" is not NULL, otherwise it
+is ignored.
+
+talloc_realloc() returns the new pointer, or NULL on failure. The call
+will fail either due to a lack of memory, or because the pointer has
+an reference (see talloc_reference()).
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_steal(const void *new_ctx, const void *ptr);
+
+The talloc_steal() function changes the parent context of a talloc
+pointer. It is typically used when the context that the pointer is
+currently a child of is going to be freed and you wish to keep the
+memory for a longer time. 
+
+The talloc_steal() function returns the pointer that you pass it. It
+does not have any failure modes.
+
+NOTE: It is possible to produce loops in the parent/child relationship
+if you are not careful with talloc_steal(). No guarantees are provided
+as to your sanity or the safety of your data if you do this.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+off_t talloc_total_size(const void *ptr);
+
+The talloc_total_size() function returns the total size in bytes used
+by this pointer and all child pointers. Mostly useful for debugging.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_report(const void *ptr, FILE *f);
+
+The talloc_report() function prints a summary report of all memory
+used by ptr. One line of report is printed for each immediate child of
+ptr, showing the total memory and number of blocks used by that child.
+
+You can pass NULL for the pointer, in which case a report is printed
+for the top level memory context.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_report_full(const void *ptr, FILE *f);
+
+This provides a more detailed report than talloc_report(). It will
+recursively print the ensire tree of memory referenced by the
+pointer. References in the tree are shown by giving the name of the
+pointer that is referenced.
+
+You can pass NULL for the pointer, in which case a report is printed
+for the top level memory context.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_enable_leak_report(void);
+
+This enables calling of talloc_report(NULL, stderr) when the program
+exits. In Samba4 this is enabled by using the --leak-report command
+line option.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void talloc_enable_leak_report_full(void);
+
+This enables calling of talloc_report_full(NULL, stderr) when the
+program exits. In Samba4 this is enabled by using the
+--leak-report-full command line option.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_zero(const void *ctx, size_t size);
+
+The talloc_zero() function is equivalent to:
+
+  ptr = talloc(ctx, size);
+  if (ptr) memset(ptr, 0, size);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_memdup(const void *ctx, const void *p, size_t size);
+
+The talloc_memdup() function is equivalent to:
+
+  ptr = talloc(ctx, size);
+  if (ptr) memcpy(ptr, p, size);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+char *talloc_strdup(const void *ctx, const char *p);
+
+The talloc_strdup() function is equivalent to:
+
+  ptr = talloc(ctx, strlen(p)+1);
+  if (ptr) memcpy(ptr, p, strlen(p)+1);
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+char *talloc_strndup(const void *t, const char *p, size_t n);
+
+The talloc_strndup() function is the talloc equivalent of the C
+library function strndup()
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+char *talloc_vasprintf(const void *t, const char *fmt, va_list ap);
+
+The talloc_vasprintf() function is the talloc equivalent of the C
+library function vasprintf()
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+char *talloc_asprintf(const void *t, const char *fmt, ...);
+
+The talloc_asprintf() function is the talloc equivalent of the C
+library function asprintf()
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+char *talloc_asprintf_append(char *s, const char *fmt, ...);
+
+The talloc_asprintf_append() function appends the given formatted 
+string to the given string. 
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_array_p(const void *ctx, type, uint_t count);
+
+The talloc_array_p() macro is equivalent to:
+
+  (type *)talloc(ctx, sizeof(type) * count);
+
+except that it provides integer overflow protection for the multiply,
+returning NULL if the multiply overflows.
+
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
+void *talloc_realloc_p(const void *ctx, void *ptr, type, uint_t count);
+
+The talloc_realloc_p() macro is equivalent to:
+
+  (type *)talloc_realloc(ctx, ptr, sizeof(type) * count);
+
+except that it provides integer overflow protection for the multiply,
+returning NULL if the multiply overflows.
+