flow::log Namespace Reference

Flow module providing logging functionality. More...

Namespaces
namespace	fs
	Short-hand for namespace boost::filesystem.

Classes
class	Async_file_logger
	An implementation of Logger that logs messages to a given file-system path but never blocks any logging thread for file I/O; suitable for heavy-duty file logging. More...
class	Buffer_logger
	An implementation of Logger that logs messages to an internal std::string buffer and provides read-only access to this buffer (for example, if one wants to write out its contents when exiting program). More...
class	Component
	A light-weight class, each object storing a component payload encoding an enum value from enum type of user's choice, and a light-weight ID of that enum type itself. More...
class	Config
	Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers is optional but encouraged; supports dynamically changing filter settings even while concurrent logging occurs. More...
class	Log_context
	Convenience class that simply stores a Logger and/or Component passed into a constructor; and returns this Logger and Component via get_logger() and get_log_component() public accessors. More...
class	Logger
	Interface that the user should implement, passing the implementing Logger into logging classes (Flow's own classes like net_flow::Node; and user's own logging classes) at construction (plus free/static logging functions). More...
struct	Msg_metadata
	Simple data store containing all of the information generated at every logging call site by flow::log, except the message itself, which is passed to Logger::do_log() assuming Logger::should_log() had returned true. More...
class	Ostream_log_msg_writer
	Utility class, each object of which wraps a given ostream and outputs discrete messages to it adorned with time stamps and other formatting such as separating newlines. More...
class	Serial_file_logger
	An internal-use implementation of Logger that logs messages to a given file-system path, blocking the calling thread while the I/O occurs, and usable safely only if logging occurs non-concurrently. More...
class	Simple_ostream_logger
	An implementation of Logger that logs messages to the given ostreams (e.g., cout or an ofstream for a file). More...
class	Thread_local_string_appender
	Internal flow::log class that facilitates a more efficient way to get util::ostream_op_to_string() behavior by allowing each thread to repeatedly reuse the structures that function creates from scratch on stack each time it is invoked; furthermore each logging entity is allotted a separate such set of structures to enable each entity to not affect the streams of other entities. More...
class	Verbosity_config
	Optional-use structure encapsulating a full set of verbosity config, such that one can parse it from a config source (like an options file) in concise form and apply it to a log::Config object. More...

Enumerations
enum class	Sev : size_t {   S_NONE = 0 , S_FATAL , S_ERROR , S_WARNING ,   S_INFO , S_DEBUG , S_TRACE , S_DATA ,   S_END_SENTINEL }
	Enumeration containing one of several message severity levels, ordered from highest to lowest. More...

Functions
void	swap (Log_context &val1, Log_context &val2)
	Log_context ADL-friendly swap: Equivalent to val1.swap(val2). More...
std::ostream &	operator<< (std::ostream &os, Sev val)
	Serializes a log::Sev to a standard output stream. More...
std::istream &	operator>> (std::istream &is, Sev &val)
	Deserializes a log::Sev from a standard input stream. More...
void	beautify_chrono_logger_this_thread (Logger *logger_ptr)
	Sets certain chrono-related formatting on the given Logger in the current thread that results in a consistent, desirable output of durations and certain types of time_points. More...
size_t	deep_size (const Msg_metadata &val)
	Estimate of memory footprint of the given value, including memory allocated on its behalf – but excluding its shallow sizeof! – in bytes. More...
std::istream &	operator>> (std::istream &is, Verbosity_config &val)
	Deserializes a Verbosity_config from a standard input stream by invoking val.parse(is). More...
std::ostream &	operator<< (std::ostream &os, const Verbosity_config &val)
	Serializes a Verbosity_config to a standard output stream. More...
bool	operator== (const Verbosity_config &val1, const Verbosity_config &val2)
	Checks for exact equality of two Verbosity_config objects. More...
bool	operator!= (const Verbosity_config &val1, const Verbosity_config &val2)
	Returns !(val1 == val2). More...

Variables
boost::thread_specific_ptr< Msg_metadata >	this_thread_sync_msg_metadata_ptr
	Thread-local Msg_metadata object used by FLOW_LOG_WITHOUT_CHECKING() for an alleged perf bonus in the synchronous-Logger code path. More...

Detailed Description

Flow module providing logging functionality.

While originally intended to be used from within the flow::net_flow module's implementation, it can be used by general user code as well. All other Flow modules expose flow::log concepts when logging is relevant, so one way or another the user is likely to have at least limited contact with flow::log if they use Flow at all. (In particular, classes and free/static functions in various Flow modules often take a Logger pointer as a constructor or function argument respectively. Hence to use such APIs one must instantiate a concrete Logger, the simplest choice being probably Simple_ostream_logger.)

(The general user [in their own, non-Flow-related code] may well prefer to use another logging system directly instead; or use boost.log. However, we humbly recommend taking a look at flow::log as a possibility, as it is highly usable, yet fast, yet small and elegant enough to ensure complete visibility into its low-level behavior, including how that affects performance. Note also that this system easily integrates with others via log::Logger interface, so one can stack this on top of perhaps a lower-level logging facility. The performance cost of this is essentially 1-2 virtual pointer lookups per log-message call.)

Log philosophy of Flow is as follows. The following is a list of goals; and for each item, sub-item(s) explain how this goal is accomplished.

• Flow is a library or set of libraries – not an application or set of applications – so logging has to flexibly work with any logging system the user code may use.
  • The Logger interface provides a way to efficiently log to an arbitrary logging output; the flow::log user must provide a simple Logger implementation that fills in what Logger::should_log() and Logger::do_log() mean.
  • To ensure maximum simplicity and usability right out of the box, flow::log provides a Logger implementation, Simple_ostream_logger, which will log to standard ostreams (including cout and cerr). The user can just use this out of the box and not even have to implement the (already trivial) Logger interface. (Probably most larger projects would need to go beyond this.)
• The logging system has to provide both error messages and informational or trace messages, and these should be easily distinguishable once passed on to the user's supplied arbitrary logging output(s).
  • There are at least 3 message severities, and they are passed to the Logger implementation, so that this can be translated as necessary into the user's logging output(s).
• The logging system is not a replacement for error reporting by Flow APIs.
  • flow::Error_code and flow::error provide at least error code reporting (in addition to log calls) where possible and applicable.
• The logging should be possible to entirely enable, entirely disable, or some reasonable point between these extremes.
  • The Logger interface's Logger::should_log() method allows filtering messages by severity (e.g., allow only WARNING messages) in any desired fashion. There is now also support for setting filtering config by "component" supplied essentially at the log message call site.
• If a given message is not ultimately logged due its severity, it should induce no performance or resource penalty (such as that incurred by assembling the message string) beyond the severity check itself.
  • The FLOW_LOG_...() macros work with the Logger::should_log()/Logger::do_log() APIs to only construct the ultimate message string if the resulting message would be logged.
• Logging should be simple without lots of extraneous features.
  • A few simple-to-use macros are the main way to log messages; see those first.
  • Every logged message just has a severity and component, which are enums, and the text of the message. File/function/line and thread ID info are auto-obtained and maintained along with the messages themselves, via the FLOW_LOG_...() macros. The user need not worry about such mundane but tricky details.
  • The FLOW_LOG_...() macros assume that get_logger() (in the context where the macro is expanded) returns a Logger* implementation, which is then used for the logging call(s) associated with the message – typically first the Logger::should_log() check and then possibly the Logger::do_log() message output. Deriving from the supplied Log_context utility class is an easy way to make this get_logger() available all over the Flow modules' implementations (and should indeed be a technique for the user's own logging code, if indeed they decided to use flow::log themselves).

log_fwd.hpp is separate from log.hpp due to C++ circular dependency nonsense. See util_fwd.hpp for some discussion on this same concept.

Todo:

There are third-party logging systems including boost.log. The to-do is to investigate using boost.log, partially or entirely replacing the manually implemented system. (To be honest I am rather partial to the simple but effective system already implemented, in particular the performance-focused trickery.) Boost is invaluable in many ways, and Flow modules use it extremely extensively, but my intuition says that in this case it may be irrelevant. Meanwhile, boost.log can, of course, be used within user's Logger interface implementation, but that's not what this to-do is about; it's about actually placing it all over Flow's implementation and APIs in place of Logger, log::Sev, etc. (Also, after a 5-minute glance at boost.log, I am noticing that some of its ideas I seem to have independently and unknowingly replicated here. It is, however, simple stuff, not rocket science.) My intuition is I hesitate to force-marry the Flow user to boost.log; the very simple Logger (abstract) interface is seemingly a gentler thing to force the Flow user into, at least relatively speaking.

Todo:

Lacking feature: message IDs. This is discussed a bit more in the log::Msg_metadata doc header.

Todo:

Lacking feature: printf()-style logging call sites, in contrast to the currently supported ostream fragment call sites. So, like, FLOW_LOG_WARNING_FMT("Hello, Mr. %s!", my_name.c_str()) could be used analogously to FLOW_LOG_WARNING("Hello, Mr " << my_name << "!!") (with std::string my_name). In terms of implementation, there is more to this than meets the eye perhaps; as ostreams potentially store (formatting) state between totally separate invocations of ostream<<, whereas printf() style functions normally do not. Internally, this likely allows for specially optimized logging code.

Todo:

Lacking feature: boost.format-style logging call sites. This is conceptually similar to – and possibly even entirely subsuming in backwards-compatible fashion – the to-do just above for printf()-style logging. Likely both to-dos should be designed/implemented in one shot. Moreover, it is possible (and would be absolutely delightful if true) that boost.format's format class can be used entirely transparently on top of the ostream-focused existing flow::log API (such as FLOW_LOG_WARNING() and buddies)!

Todo:

Lacking feature: log message rate-limiting. This could actually mean a few things. One is being able to rate-limit the messages produced a given log call site per unit time; this might be the minimum for this to-do. This could be done by a dirty macro hack; or it could be done in more civilized fashion (while minding perf) by configuring it by message ID (but message IDs are optional and not implemented as of this writing anyway). Another rate-limiting thing is general controls on frequency of logging; though arguably that is more applicable to individual Logger implementations rather than as a mandatory-general mechanism.

Todo:

Lacking feature: compiler hints for optimizing away log filter checks. This is inspired by a certain other proprietary logging API in which we noticed attempts to give hints to the compiler as to how likely or unlikely the verbosity check is to return a true or false value, seemingly so that the compiler might optimize out the check and hence the entire log statement in some conditions; or always log others and skip the check in the optimized code. I omit any details here, but that's the general idea. I don't know how effective this is given that verbosities are configurable dynamically potentially; but look into it.

Enumeration Type Documentation

Sev

enum class flow::log::Sev : size_t

strong

Enumeration containing one of several message severity levels, ordered from highest to lowest.

Generally speaking, volume/verbosity is inversely proportional to severity, though this is not enforced somehow.

As the underlying type is size_t, and the values are guaranteed to be 0, 1, ... going from lowest to highest verbosity (highest to lowest severity), you may directly use log::Sev values to index into arrays that arrange one-to-one values in the same order.

The supplied ostream<< operator, together with this enum, is suitable for util::istream_to_enum(). ostream>> operator is built on the latter and is also supplied. Hence I/O of Sev is available out of the box; this enables parsing in boost.program_options among other things.

Formal semantics of log::Sev

From the point of view of all Flow code, the formal semantics of log::Sev are as follows:

• Sev::S_NONE has value 0 and is a start-sentinel.
• Non-sentinel severities follow, starting at 1 with no gaps.
  • Ordinal comparison is meaningful; lower is termed as more severe/less verbose; higher is therefore termed as less severe/more verbose. These are, formally, only terminology.
• Sev::S_END_SENTINEL is the end-sentinel.
• Sev::S_WARNING is and always shall be considered the least severe "abnormal" condition. Any additional "abnormal" conditions shall always have more-severe (numerically lower) values. As of this writing, within flow::log Flow module, this has only one practical meaning (but that could change): Simple_ostream_logger directs WARNING-and-more-severe messages to its configured error stream (os_for_err); while all the rest are directed to its configured other (regular) stream (os). (For example these might be set to std::cerr and std::cout respectively; or both to std::cout.)

Informal semantics of log::Sev

The following are not enforced by any code logic in the flow::log Flow module. However we suggest users understand and consider these, as a haphazard approach to severity selection for a given log call site can cause significant operational problems in a high-volume production environment (at least).

Generally, there are 2 types of log call sites: ones inside Flow; and ones outside (user code). The former (inside Flow) are deliberately minimalistic, so as to use as few severities as possible and thus make severity selection maximally simple and, importantly, unambiguous.

However, logging code outside Flow may require more sophisticated in-use severity sets. Informally we recommend keeping it as simple as inside-Flow's scheme... if it is sufficient. If it is not sufficient, the other severities may also be used.

The log::Sev values used inside Flow (the minimalistic set) are as follows. (Brief meanings accompany them; see their individual doc headers for more detail.) From most to least severe:

• Sev::S_FATAL: the program will abort shortly due to the condition that is being logged. Usually an assert(false) and/or std::abort() follows. If you've disabled abort-on-assertion-trip (NDEBUG is defined), and there is no std::abort() or equivalent, then the program may continue, but subsequent behavior is undefined.
• Sev::S_WARNING: abnormal condition is being logged, and its aggregate volume is not high enough to be classified as TRACE instead to avoid perf impact. (Other than possibly FATAL – used in extreme cases – no other abnormal-condition Sev are in-use inside Flow nor will they be.)
• Sev::S_INFO: non-abnormal condition is being logged, and it's not so frequent in practice that enabling this log level shall adversely impact performance.
• Sev::S_TRACE: like INFO, but enabling this log level can adversely impact performance. However, entire dumps of potentially large (in aggregate) data (such as packet contents) being processed are not included.
• Sev::S_DATA: like TRACE, but entire contents of potentially large (in aggregate) data (such as packet contents) being processed are included in the message, which may lead to particularly large log output (if enabled).

As of this writing the following log::Sev values are available for use (by user/non-Flow code) beyond the above minimalistic set. Potential subjective meanings are included, but user code can use whatever conventions that suit them best.

• Sev::S_ERROR: A non-FATAL abnormal condition subjectively more severe than WARNING.
• Sev::S_DEBUG: A non-abnormal condition with, perhaps, non-perf-affecting verbosity (a-la INFO) but subjectively of less interest to a human glancing over a large-ish log snippet (a-la TRACE).

Enumerator
S_NONE	Sentinel log level that must not be specified for any actual message (at risk of undefined behavior such as assert failure); but can be used for sentinel purposes such as specifying a log filter wherein no messages (not even Sev::S_WARNING ones) are shown.
S_FATAL	Message indicates a "fatally bad" condition, such that the program shall imminently abort, typically due to immediately-following assert(false) and possibly std::abort(); or if aborts are disabled (such as via defining NDEBUG), and there is no explicit std::abort() or equivalent, then further program behavior is undefined. Note This severity is part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section.
S_ERROR	Message indicates a "bad" condition with "worse" impact than that of Sev::S_WARNING. Note If it's "bad" but frequent enough for TRACE, we strongly recommend to make it TRACE – not ERROR. This severity is NOT part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section. Informally we recommend projects do not use it unless it is holistically necessary.
S_WARNING	Message indicates a "bad" condition that is not frequent enough to be of severity Sev::S_TRACE. These typically should occur with less frequency than INFO messages; however, it's not a hard rule. Note If it's "bad" but frequent enough for TRACE, it's TRACE by definition – not WARNING. This severity is part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section.
S_INFO	Message indicates a not-"bad" condition that is not frequent enough to be of severity Sev::S_TRACE. Note That is, it's identical to WARNING except doesn't represent a "bad" situation. This severity is part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section.
S_DEBUG	Message indicates a condition with, perhaps, no significant perf impact if enabled (like Sev::S_INFO) but of subjectively less interest to a human reader than INFO (hence, like Sev::S_TRACE in that respect). Note This severity is NOT part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section. Informally we recommend projects do not use it unless it is holistically necessary.
S_TRACE	Message indicates any condition that may occur with great frequency (thus verbose if logged). The line between Sev::S_INFO or Sev::S_WARNING and TRACE is as follows: The former is not allowed to classify messages such that in realistic scenarios they would degrade performance, from processor cycles or log file I/O. See also Sev::S_DATA for an even-more-verbose severity, when it's frequent and potentially large in size in aggregate. Warning One MUST be able to set max severity level to INFO and confidently count that logging will not affect performance. Note This severity is part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section.
S_DATA	Message satisfies Sev::S_TRACE description AND contains variable-length structure (like packet, file) dumps. If these are allowed to be logged, resulting log file might be roughly similar to (or even larger than) the data being transmitted by the logging code. Packet dumps are obvious examples. Note that just because it's a variable-length structure dump doesn't mean it's for Sev::S_DATA severity. If it's not frequent, it's fine for it to even be INFO. E.g., if I decide to dump every RST packet, it's probably okay as Sev::S_INFO, since RSTs are practically rare; that need not be Sev::S_DATA. On the other hand, if I dump every DATA packet as anything except Sev::S_DATA severity, then I should be arrested and convicted. When this level is disabled, consider logging a TRACE message instead but summarize the contents you'd dump; e.g., log a hash and/or a size and/or the few first bytes instead of the entire thing. Something is often better than nothing (but nothing is still safer than the whole thing). Note This severity is part of the minimalistic (in-use within Flow) severity set as discussed in log::Sev doc header, "Informal semantics" section.
S_END_SENTINEL	Not an actual value but rather stores the highest numerical payload, useful for validity checks.

Definition at line 223 of file log_fwd.hpp.

Function Documentation

beautify_chrono_logger_this_thread()

void flow::log::beautify_chrono_logger_this_thread

(

Logger *

logger_ptr

)

Sets certain chrono-related formatting on the given Logger in the current thread that results in a consistent, desirable output of durations and certain types of time_points.

The effect is that of util::beautify_chrono_ostream().

See also

flow::async in which new threads are set to use this formatting automatically. However you'll want to do this explicitly for the startup thread.

Parameters

logger_ptr

The Logger to affect in this thread. Null is allowed; results in no-op.

Definition at line 278 of file log.cpp.

References flow::util::beautify_chrono_ostream(), and flow::log::Logger::this_thread_ostream().

Referenced by flow::async::Task_qing_thread::Task_qing_thread(), and flow::net_flow::Node::this_thread_init_logger_setup().

Here is the call graph for this function:

Here is the caller graph for this function:

deep_size()

size_t flow::log::deep_size

(

const Msg_metadata &

val

)

Estimate of memory footprint of the given value, including memory allocated on its behalf – but excluding its shallow sizeof! – in bytes.

Parameters

val

Value.

Returns

See above.

Definition at line 288 of file log.cpp.

References deep_size(), flow::util::deep_size(), and flow::log::Msg_metadata::m_call_thread_nickname.

Referenced by flow::log::Async_file_logger::deep_size(), and deep_size().

Here is the call graph for this function:

Here is the caller graph for this function:

operator!=()

bool operator!=	(	const Verbosity_config &	val1,
		const Verbosity_config &	val2
	)

Returns !(val1 == val2).

Parameters

val1	Object to compare.
val2	Object to compare.

Returns

See above.

Definition at line 236 of file verbosity_config.cpp.

operator<<() [1/2]

std::ostream & operator<<	(	std::ostream &	os,
		const Verbosity_config &	val
	)

Serializes a Verbosity_config to a standard output stream.

Parameters

os	Stream to which to serialize.
val	Value to serialize.

Returns

os.

Definition at line 210 of file verbosity_config.cpp.

operator<<() [2/2]

std::ostream & flow::log::operator<<	(	std::ostream &	os,
		Sev	val
	)

Serializes a log::Sev to a standard output stream.

The output string is compatible with the reverse istream>> operator.

Parameters

os	Stream to which to serialize.
val	Value to serialize.

Returns

os.

Definition at line 249 of file log.cpp.

References S_DATA, S_DEBUG, S_END_SENTINEL, S_ERROR, S_FATAL, S_INFO, S_NONE, S_TRACE, and S_WARNING.

operator==()

bool operator==	(	const Verbosity_config &	val1,
		const Verbosity_config &	val2
	)

Checks for exact equality of two Verbosity_config objects.

(Note that this is not maximally clever, in that if (val1 == val2), then they definitely produce the same Config all else being equal; but if the reverse is true, then it is possible they differently expressed values producing the same actual result. E.g., something like ALL:INFO differs from ALL:WARN;ALL:INFO – yet they have the same effect. However component names are normalized internally when parsing, so that won't produce false inequality.)

Only Verbosity_config::component_sev_pairs() is significant in this comparison; last_result_message() is not.

Parameters

val1	Object to compare.
val2	Object to compare.

Returns

true if definitely equal; false if possibly not equal.

Definition at line 231 of file verbosity_config.cpp.

Referenced by flow::log::Verbosity_config::operator!=().

Here is the caller graph for this function:

operator>>() [1/2]

std::istream & flow::log::operator>>	(	std::istream &	is,
		Sev &	val
	)

Deserializes a log::Sev from a standard input stream.

Reads up to but not including the next non-alphanumeric-or-underscore character; the resulting string is then mapped to a log::Sev. If none is recognized, Sev::S_NONE is the result. The recognized values are:

• "0", "1", ...: Corresponds to the int conversion of that log::Sev (e.g., 0 being NONE).
• Case-insensitive encoding of the non-S_-prefix part of the actual log::Sev member; e.g., "warning" (or "Warning" or "WARNING" or...) for S_WARNING. This enables a few key things to work, including parsing from config file/command line via and conversion from string via boost::lexical_cast.

Parameters

is	Stream from which to deserialize.
val	Value to set.

Returns

is.

Definition at line 269 of file log.cpp.

References flow::util::istream_to_enum(), S_END_SENTINEL, and S_NONE.

Here is the call graph for this function:

operator>>() [2/2]

std::istream & operator>>	(	std::istream &	is,
		Verbosity_config &	val
	)

Deserializes a Verbosity_config from a standard input stream by invoking val.parse(is).

val.last_result_message() can be used to glean the success or failure of this operation.

Parameters

is	Stream from which to deserialize.
val	Value to set.

Returns

is.

Definition at line 204 of file verbosity_config.cpp.

swap()

void flow::log::swap	(	Log_context &	val1,
		Log_context &	val2
	)

Log_context ADL-friendly swap: Equivalent to val1.swap(val2).

Parameters

val1	Object.
val2	Object.

Definition at line 242 of file log.cpp.

References flow::log::Log_context::swap().

Referenced by flow::log::Log_context::operator=(), and flow::log::Log_context::swap().

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

this_thread_sync_msg_metadata_ptr

boost::thread_specific_ptr< Msg_metadata > flow::log::this_thread_sync_msg_metadata_ptr

Thread-local Msg_metadata object used by FLOW_LOG_WITHOUT_CHECKING() for an alleged perf bonus in the synchronous-Logger code path.

It is allocated at the first log call site for a given thread; and auto-deallocated when thread exits.

Definition at line 28 of file log.cpp.