Export sbuf_drain to orchestrate lock and drain action

While exporting large amounts of data to a sysctl
request, datastructures may need to be locked.

Exporting the sbuf_drain function allows the

Export sbuf_drain to orchestrate lock and drain action

While exporting large amounts of data to a sysctl
request, datastructures may need to be locked.

Exporting the sbuf_drain function allows the
coordination between drain events and held
locks, to avoid stalls.

PR: 254333
Reviewed By: jhb
MFC after: 2 weeks
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D29481

(cherry picked from commit c804c8f2c58ba42d476de07fbceff9ac4dd95f0e)

show more ...

sbuf(9): Add sbuf_nl_terminate() API

The API is used to gracefully terminate text line(s) with a single \n. If
the formatted buffer was empty or already ended in \n, it is unmodified.
Otherwise, a

sbuf(9): Add sbuf_nl_terminate() API

The API is used to gracefully terminate text line(s) with a single \n. If
the formatted buffer was empty or already ended in \n, it is unmodified.
Otherwise, a newline character is appended to it. The API, like other
sbuf-modifying routines, is only valid while the sbuf is not FINISHED.

Reviewed by: rlibby
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D21030

show more ...

sbuf(9): Add NOWAIT dynamic buffer extension mode

The goal is to avoid some kinds of low-memory deadlock when formatting
heap-allocated buffers.

Reviewed by: vangyzen
Sponsored by: Dell EMC Isilon

sbuf(9): Add NOWAIT dynamic buffer extension mode

The goal is to avoid some kinds of low-memory deadlock when formatting
heap-allocated buffers.

Reviewed by: vangyzen
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D21015

show more ...

Optimize kern.geom.conf* sysctls.

On large systems those sysctls may generate megabytes of output. Before
this change sbuf(9) code was resizing buffer by 4KB each time many times,
generating tons o

Optimize kern.geom.conf* sysctls.

On large systems those sysctls may generate megabytes of output. Before
this change sbuf(9) code was resizing buffer by 4KB each time many times,
generating tons of TLB shootdowns. Unfortunately in this case existing
sbuf_new_for_sysctl() mechanism, supposed to help with this issue, is not
applicable, since all the sbuf writes are done in different kernel thread.

This change improves situation in two ways:
- on first sysctl call, not providing any output buffer, it sets special
sbuf drain function, just counting the data and so not needing big buffer;
- on second sysctl call it uses as initial buffer size value saved on
previous call, so that in most cases there will be no reallocation, unless
GEOM topology changed significantly.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

show more ...

device_printf: Use sbuf for more coherent prints on SMP

device_printf does multiple calls to printf allowing other console messages to
be inserted between the device name, and the rest of the messag

device_printf: Use sbuf for more coherent prints on SMP

device_printf does multiple calls to printf allowing other console messages to
be inserted between the device name, and the rest of the message. This change
uses sbuf to compose to two into a single buffer, and prints it all at once.

It exposes an sbuf drain function (drain-to-printf) for common use.

Update documentation to match; some unit tests included.

Submitted by: jmg
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D16690

show more ...

sys/sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

sys/sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

Implement simple record boundary tracking in sbuf(9) to avoid record splitting
during drain operations. When an sbuf is configured to use this feature by way
of the SBUF_DRAINTOEOR sbuf_new() flag, t

Implement simple record boundary tracking in sbuf(9) to avoid record splitting
during drain operations. When an sbuf is configured to use this feature by way
of the SBUF_DRAINTOEOR sbuf_new() flag, top-level sections started with
sbuf_start_section() create a record boundary marker that is used to avoid
flushing partial records.

Reviewed by: cem,imp,wblock
MFC after: 2 weeks
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D8536

show more ...

Add prototype for sbuf_putbuf()

Sponsored by: Netflix

Add support for reading MAM attributes to camcontrol(8) and libcam(3).

MAM is Medium Auxiliary Memory and is most commonly found as flash
chips on tapes.

This includes support for reading attribute

Add support for reading MAM attributes to camcontrol(8) and libcam(3).

MAM is Medium Auxiliary Memory and is most commonly found as flash
chips on tapes.

This includes support for reading attributes and decoding most
known attributes, but does not yet include support for writing
attributes or reporting attributes in XML format.

libsbuf/Makefile:
Add subr_prf.c for the new sbuf_hexdump() function. This
function is essentially the same function.

libsbuf/Symbol.map:
Add a new shared library minor version, and include the
sbuf_hexdump() function.

libsbuf/Version.def:
Add version 1.4 of the libsbuf library.

libutil/hexdump.3:
Document sbuf_hexdump() alongside hexdump(3), since it is
essentially the same function.

camcontrol/Makefile:
Add attrib.c.

camcontrol/attrib.c:
Implementation of READ ATTRIBUTE support for camcontrol(8).

camcontrol/camcontrol.8:
Document the new 'camcontrol attrib' subcommand.

camcontrol/camcontrol.c:
Add the new 'camcontrol attrib' subcommand.

camcontrol/camcontrol.h:
Add a function prototype for scsiattrib().

share/man/man9/sbuf.9:
Document the existence of sbuf_hexdump() and point users to
the hexdump(3) man page for more details.

sys/cam/scsi/scsi_all.c:
Add a table of known attributes, text descriptions and
handler functions.

Add a new scsi_attrib_sbuf() function along with a number
of other related functions that help decode attributes.

scsi_attrib_ascii_sbuf() decodes ASCII format attributes.

scsi_attrib_int_sbuf() decodes binary format attributes, and
will pass them off to scsi_attrib_hexdump_sbuf() if they're
bigger than 8 bytes.

scsi_attrib_vendser_sbuf() decodes the vendor and drive
serial number attribute.

scsi_attrib_volcoh_sbuf() decodes the Volume Coherency
Information attribute that LTFS writes out.

sys/cam/scsi/scsi_all.h:
Add a number of attribute-related structure definitions and
other defines.

Add function prototypes for all of the functions added in
scsi_all.c.

sys/kern/subr_prf.c:
Add a new function, sbuf_hexdump(). This is the same as
the existing hexdump(9) function, except that it puts the
result in an sbuf.

This also changes subr_prf.c so that it can be compiled in
userland for includsion in libsbuf.

We should work to change this so that the kernel hexdump
implementation is a wrapper around sbuf_hexdump() with a
statically allocated sbuf with a drain. That will require
a drain function that goes to the kernel printf() buffer
that can take a non-NUL terminated string as input.
That is because an sbuf isn't NUL-terminated until it is
finished, and we don't want to finish it while we're still
using it.

We should also work to consolidate the userland hexdump and
kernel hexdump implemenatations, which are currently
separate. This would also mean making applications that
currently link in libutil link in libsbuf.

sys/sys/sbuf.h:
Add the prototype for sbuf_hexdump(), and add another copy
of the hexdump flag values if they aren't already defined.

Ideally the flags should be defined in one place but the
implemenation makes it difficult to do properly. (See
above.)

Sponsored by: Spectra Logic Corporation
MFC after: 1 week

show more ...

Add a new flag, SBUF_INCLUDENUL, and new get/set/clear functions for flags.

The SBUF_INCLUDENUL flag causes the nulterm byte at the end of the string
to be counted in the length of the data. If cop

Add a new flag, SBUF_INCLUDENUL, and new get/set/clear functions for flags.

The SBUF_INCLUDENUL flag causes the nulterm byte at the end of the string
to be counted in the length of the data. If copying the data using the
sbuf_data() and sbuf_len() functions, or if writing it automatically with
a drain function, the net effect is that the nulterm byte is copied along
with the rest of the data.

show more ...

Unbreak the ABI by reverting r268494 until the compat shims are provided

Implement Short/Small String Optimization in SBUF(9) and change lengths and
positions in the API from ssize_t and int to size_t.

CR: D388
Approved by: des, bapt

Add sbuf_start_section() and sbuf_end_section() functions, which can
be used for automatic section alignment.

Discussed with: kib
Reviewed by: kib
MFC after: 1 month

Change the length quantities of sbufs to be ssize_t rather than int.

Constify a couple of arguments.

Replace sbuf_overflowed() with sbuf_error(), which returns any error
code associated with overflow or with the drain function. While this
function is not expected to be used often, it produces more

Replace sbuf_overflowed() with sbuf_error(), which returns any error
code associated with overflow or with the drain function. While this
function is not expected to be used often, it produces more information
in the form of an errno that sbuf_overflowed() did.

show more ...

Add drain functionality to sbufs. The drain is a function that is
called when the sbuf internal buffer is filled. For kernel sbufs with a
drain, the internal buffer will never be expanded. For use

Add drain functionality to sbufs. The drain is a function that is
called when the sbuf internal buffer is filled. For kernel sbufs with a
drain, the internal buffer will never be expanded. For userland sbufs
with a drain, the internal buffer may still be expanded by
sbuf_[v]printf(3).

Sbufs now have three basic uses:
1) static string manipulation. Overflow is marked.
2) dynamic string manipulation. Overflow triggers string growth.
3) drained string manipulation. Overflow triggers draining.

In all cases the manipulation is 'safe' in that overflow is detected and
managed.

Reviewed by: phk (the previous version)

show more ...

Add sbuf_new_auto as a shortcut for the very common case of creating a
completely dynamic sbuf.

Obtained from: Varnish
MFC after: 2 weeks

Switch to simplified BSD license (with phk's approval), plus whitespace
and style(9) cleanup.

Have sbuf_bcat() and sbuf_bcpy() take a const void * instead of a
const char *, since callers are likely to pass in pointers to all
kinds of structs and whatnot.

Add the new function "sbuf_done()" which returns non-zero if the sbuf is
finished.

This allows sbufs to be used for request/response scenarioes without
needing additional communication flags.

Spons

Add the new function "sbuf_done()" which returns non-zero if the sbuf is
finished.

This allows sbufs to be used for request/response scenarioes without
needing additional communication flags.

Sponsored by: DARPA & NAI Labs.

show more ...

o Merge <machine/ansi.h> and <machine/types.h> into a new header
called <machine/_types.h>.
o <machine/ansi.h> will continue to live so it can define MD clock
macros, which are only MD because of

o Merge <machine/ansi.h> and <machine/types.h> into a new header
called <machine/_types.h>.
o <machine/ansi.h> will continue to live so it can define MD clock
macros, which are only MD because of gratuitous differences between
architectures.
o Change all headers to make use of this. This mainly involves
changing:
#ifdef _BSD_FOO_T_
typedef _BSD_FOO_T_ foo_t;
#undef _BSD_FOO_T_
#endif
to:
#ifndef _FOO_T_DECLARED
typedef __foo_t foo_t;
#define _FOO_T_DECLARED
#endif

Concept by: bde
Reviewed by: jake, obrien

show more ...

Replace spaces after #defines with tabs; this makes all #defines
consistent in their adherence with style(9).

* Implement SBUF_AUTOEXTEND flag; sbufs created with this flag are
automatically extended to prevent overflow.
* Added sbuf_vprintf(); sbuf_printf() is now just a wrapper around
sbuf_vprintf()

* Implement SBUF_AUTOEXTEND flag; sbufs created with this flag are
automatically extended to prevent overflow.
* Added sbuf_vprintf(); sbuf_printf() is now just a wrapper around
sbuf_vprintf().
* Include <stdio.h> and <string.h> when building libsbuf to silence
WARNS=4 warnings.

Reviewed by: des

show more ...

Reduce namespace pollution by removing argument names from prototypes.

Add a couple of API functions I need for my pseudofs WIP. Documentation
will follow when I've decided whether to keep this API or ditch it in
favor of something slightly more subtle.