zl程序教程

您现在的位置是:首页 >  数据库

当前栏目

Mysql:Server Logs:mysqld服务器的各种日志

mysqlserver服务器日志 各种 Logs Mysqld
2023-09-27 14:24:27 时间

比较新的知识点:DDL log,以前没注意到

5.4 MySQL Server Logs

MySQL Server has several logs that can help you find out what activity is taking place.

Log TypeInformation Written to Log
Error log Problems encountered starting, running, or stopping mysqld
General query log Established client connections and statements received from clients
Binary log Statements that change data (also used for replication)
Relay log Data changes received from a replication master server
Slow query log Queries that took more than long_query_time seconds to execute
DDL log (metadata log) Metadata operations performed by DDL statements

By default, no logs are enabled, except the error log on Windows. (The DDL log is always created when required, and has no user-configurable options; see Section 5.4.6, “The DDL Log”.) The following log-specific sections provide information about the server options that enable logging.

By default, the server writes files for all enabled logs in the data directory. You can force the server to close and reopen the log files (or in some cases switch to a new log file) by flushing the logs. Log flushing occurs when you issue a FLUSH LOGS statement; execute mysqladmin with a flush-logs or refresh argument; or execute mysqldump with a --flush-logs or --master-data option. See Section 13.7.8.3, “FLUSH Statement”, Section 4.5.2, “mysqladmin — Client for Administering a MySQL Server”, and Section 4.5.4, “mysqldump — A Database Backup Program”. In addition, the binary log is flushed when its size reaches the value of the max_binlog_size system variable.

You can control the general query and slow query logs during runtime. You can enable or disable logging, or change the log file name. You can tell the server to write general query and slow query entries to log tables, log files, or both. For details, see Section 5.4.1, “Selecting General Query Log and Slow Query Log Output Destinations”, Section 5.4.3, “The General Query Log”, and Section 5.4.5, “The Slow Query Log”.

The relay log is used only on slave replication servers, to hold data changes from the master server that must also be made on the slave. For discussion of relay log contents and configuration, see Section 17.2.4.1, “The Slave Relay Log”.

For information about log maintenance operations such as expiration of old log files, see Section 5.4.7, “Server Log Maintenance”.

For information about keeping logs secure, see Section 6.1.2.3, “Passwords and Logging”.

5.4.1 Selecting General Query Log and Slow Query Log Output Destinations

MySQL Server provides flexible control over the destination of output written to the general query log and the slow query log, if those logs are enabled. Possible destinations for log entries are log files or the general_log and slow_log tables in the mysql system database. File output, table output, or both can be selected.

Log Control at Server Startup

The log_output system variable specifies the destination for log output. Setting this variable does not in itself enable the logs; they must be enabled separately.

  • If log_output is not specified at startup, the default logging destination is FILE.

  • If log_output is specified at startup, its value is a list one or more comma-separated words chosen from TABLE (log to tables), FILE (log to files), or NONE (do not log to tables or files). NONE, if present, takes precedence over any other specifiers.

The general_log system variable controls logging to the general query log for the selected log destinations. If specified at server startup, general_log takes an optional argument of 1 or 0 to enable or disable the log. To specify a file name other than the default for file logging, set the general_log_file variable. Similarly, the slow_query_log variable controls logging to the slow query log for the selected destinations and setting slow_query_log_file specifies a file name for file logging. If either log is enabled, the server opens the corresponding log file and writes startup messages to it. However, further logging of queries to the file does not occur unless the FILE log destination is selected.

Examples:

  • To write general query log entries to the log table and the log file, use --log_output=TABLE,FILE to select both log destinations and --general_log to enable the general query log.

  • To write general and slow query log entries only to the log tables, use --log_output=TABLE to select tables as the log destination and --general_log and --slow_query_log to enable both logs.

  • To write slow query log entries only to the log file, use --log_output=FILE to select files as the log destination and --slow_query_log to enable the slow query log. In this case, because the default log destination is FILE, you could omit the log_output setting.

Log Control at Runtime

The system variables associated with log tables and files enable runtime control over logging:

  • The log_output variable indicates the current logging destination. It can be modified at runtime to change the destination.

  • The general_log and slow_query_log variables indicate whether the general query log and slow query log are enabled (ON) or disabled (OFF). You can set these variables at runtime to control whether the logs are enabled.

  • The general_log_file and slow_query_log_file variables indicate the names of the general query log and slow query log files. You can set these variables at server startup or at runtime to change the names of the log files.

  • To disable or enable general query logging for the current session, set the session sql_log_off variable to ON or OFF. (This assumes that the general query log itself is enabled.)

Log Table Benefits and Characteristics

The use of tables for log output offers the following benefits:

  • Log entries have a standard format. To display the current structure of the log tables, use these statements:

    SHOW CREATE TABLE mysql.general_log;
    SHOW CREATE TABLE mysql.slow_log;
  • Log contents are accessible through SQL statements. This enables the use of queries that select only those log entries that satisfy specific criteria. For example, to select log contents associated with a particular client (which can be useful for identifying problematic queries from that client), it is easier to do this using a log table than a log file.

  • Logs are accessible remotely through any client that can connect to the server and issue queries (if the client has the appropriate log table privileges). It is not necessary to log in to the server host and directly access the file system.

The log table implementation has the following characteristics:

  • In general, the primary purpose of log tables is to provide an interface for users to observe the runtime execution of the server, not to interfere with its runtime execution.

  • CREATE TABLE, ALTER TABLE, and DROP TABLE are valid operations on a log table. For ALTER TABLE and DROP TABLE, the log table cannot be in use and must be disabled, as described later.

  • By default, the log tables use the CSV storage engine that writes data in comma-separated values format. For users who have access to the .CSV files that contain log table data, the files are easy to import into other programs such as spreadsheets that can process CSV input.

    The log tables can be altered to use the MyISAM storage engine. You cannot use ALTER TABLE to alter a log table that is in use. The log must be disabled first. No engines other than CSV or MyISAM are legal for the log tables.

    Log Tables and Too many open files” Errors.  If you select TABLE as a log destination and the log tables use the CSV storage engine, you may find that disabling and enabling the general query log or slow query log repeatedly at runtime results in a number of open file descriptors for the .CSV file, possibly resulting in a Too many open files” error. To work around this issue, execute FLUSH TABLES or ensure that the value of open_files_limit is greater than the value of table_open_cache_instances.

  • To disable logging so that you can alter (or drop) a log table, you can use the following strategy. The example uses the general query log; the procedure for the slow query log is similar but uses the slow_log table and slow_query_log system variable.

    SET @old_log_state = @@GLOBAL.general_log;
    SET GLOBAL general_log = 'OFF';
    ALTER TABLE mysql.general_log ENGINE = MyISAM;
    SET GLOBAL general_log = @old_log_state;
  • TRUNCATE TABLE is a valid operation on a log table. It can be used to expire log entries.

  • RENAME TABLE is a valid operation on a log table. You can atomically rename a log table (to perform log rotation, for example) using the following strategy:

    USE mysql;
    DROP TABLE IF EXISTS general_log2;
    CREATE TABLE general_log2 LIKE general_log;
    RENAME TABLE general_log TO general_log_backup, general_log2 TO general_log;
  • CHECK TABLE is a valid operation on a log table.

  • LOCK TABLES cannot be used on a log table.

  • INSERT, DELETE, and UPDATE cannot be used on a log table. These operations are permitted only internally to the server itself.

  • FLUSH TABLES WITH READ LOCK and the state of the read_only system variable have no effect on log tables. The server can always write to the log tables.

  • Entries written to the log tables are not written to the binary log and thus are not replicated to slave servers.

  • To flush the log tables or log files, use FLUSH TABLES or FLUSH LOGS, respectively.

  • Partitioning of log tables is not permitted.

  • A mysqldump dump includes statements to recreate those tables so that they are not missing after reloading the dump file. Log table contents are not dumped.

5.4.2 The Error Log

This section discusses how to configure the MySQL server for logging of diagnostic messages to the error log. For information about selecting the error message character set or language, see Section 10.6, “Error Message Character Set”, or Section 10.12, “Setting the Error Message Language”.

The error log contains a record of mysqld startup and shutdown times. It also contains diagnostic messages such as errors, warnings, and notes that occur during server startup and shutdown, and while the server is running. For example, if mysqld notices that a table needs to be automatically checked or repaired, it writes a message to the error log.

On some operating systems, the error log contains a stack trace if mysqld exits abnormally. The trace can be used to determine where mysqld exited. See Section 29.5, “Debugging and Porting MySQL”.

If used to start mysqld, mysqld_safe may write messages to the error log. For example, when mysqld_safe notices abnormal mysqld exits, it restarts mysqld and writes a mysqld restarted message to the error log.

The following sections discuss aspects of configuring error logging.

5.4.2.1 Error Log Component Configuration

In MySQL 8.0, error logging uses the MySQL component architecture described at Section 5.5, “MySQL Server Components”. The error log subsystem consists of components that perform log event filtering and writing, as well as a system variable that configures which components to enable to achieve the desired logging result.

This section discusses how to select components for error logging. For instructions specific to log filters, see Section 5.4.2.3, “Types of Error Log Filtering”. For instructions specific to the JSON and system log writers, see Section 5.4.2.6, “Error Logging in JSON Format”, and Section 5.4.2.7, “Error Logging to the System Log”. For additional details about all available log components, see Section 5.5.3, “Error Log Components”.

Component-based error logging offers these features:

  • Log events can be filtered by filter components to affect the information available for writing.

  • Log events are output by sink (writer) components. Multiple sink components can be enabled, to write error log output to multiple destinations.

  • Built-in filter and writer components combine to implement the default error log format.

  • A loadable writer enables logging in JSON format.

  • A loadable writer enables logging to the system log.

  • System variables control which log components to enable and how each component operates.

The log_error_services system variable controls which log components to enable for error logging. The variable may contain a list with 0, 1, or many elements. In the latter case, elements may be delimited by semicolon or (as of MySQL 8.0.12) comma, optionally followed by space. A given setting cannot use both semicolon and comma separators. Component order is significant because the server executes components in the order listed.

By default, log_error_services has this value:

mysql> SELECT @@GLOBAL.log_error_services;
+----------------------------------------+
| @@GLOBAL.log_error_services            |
+----------------------------------------+
| log_filter_internal; log_sink_internal |
+----------------------------------------+

That value indicates that log events first pass through the built-in filter component, log_filter_internal, then through the built-in log writer component, log_sink_internal. A filter modifies log events seen by components named later in the log_error_services value. A sink is a destination for log events. Typically, a sink processes log events into log messages that have a particular format and writes these messages to its associated output, such as a file or the system log.

Note

Assigning log_error_services a value that contains no writer components causes no log output to be written from that point.

The final component in the log_error_services value should be a writer. If the final component is a filter, the component has no effect because the filtered events are not sent to any writer.

The combination of log_filter_internal and log_sink_internal implements the default error log filtering and output behavior. The action of these components is affected by other server options and system variables:

To change the set of log components used for error logging, load components as necessary and modify the log_error_services value. Adding or removing log components is subject to these constraints:

For example, to use the system log writer (log_sink_syseventlog) instead of the default writer (log_sink_internal), first load the writer component, then modify the log_error_services value:

INSTALL COMPONENT 'file://component_log_sink_syseventlog';
SET GLOBAL log_error_services = 'log_filter_internal; log_sink_syseventlog';
Note

The URN to use for loading a log component with INSTALL COMPONENT is the component name prefixed with file://component_. For example, for the log_sink_syseventlog component, the corresponding URN is file://component_log_sink_syseventlog.

It is possible to configure multiple log writers, which enables sending output to multiple destinations. To enable the system log writer in addition to (rather than instead of) the default writer, set the log_error_services value like this:

SET GLOBAL log_error_services = 'log_filter_internal; log_sink_internal; log_sink_syseventlog';

To revert to using only the default writer and unload the system log writer, execute these statements:

SET GLOBAL log_error_services = 'log_filter_internal; log_sink_internal;
UNINSTALL COMPONENT 'file://component_log_sink_syseventlog';

To configure a log component to be enabled at each server startup, use this procedure:

  1. If the component is loadable, load it at runtime using INSTALL COMPONENT. Loading the component registers it in the mysql.component system table so that the server loads it automatically for subsequent startups.

  2. Set the log_error_services value at startup to include the component name. Set the value either in the server my.cnf file, or use SET PERSIST, which sets the value for the running MySQL instance and also saves the value to be used for subsequent server restarts; see Section 13.7.6.1, “SET Syntax for Variable Assignment”. A value set in my.cnf takes effect at the next restart. A value set using SET PERSIST takes effect immediately, and for subsequent restarts.

Suppose that you want to configure, for every server startup, use of the JSON log writer (log_sink_json) in addition to the built-in log filter and writer (log_filter_internal, log_sink_internal). First load the JSON writer if it is not loaded:

INSTALL COMPONENT 'file://component_log_sink_json';

Then set log_error_services to take effect at server startup. You can set it in my.cnf:

[mysqld]
log_error_services='log_filter_internal; log_sink_internal; log_sink_json'

Or you can set it using SET PERSIST:

SET PERSIST log_error_services = 'log_filter_internal; log_sink_internal; log_sink_json';

The order of components named in log_error_services is significant, particularly with respect to the relative order of filters and writers. Consider this log_error_services value:

log_filter_internal; log_sink_1; log_sink_2

In this case, log events pass to the built-in filter, then to the first writer, then to the second writer. Both writers receive the filtered log events.

Compare that to this log_error_services value:

log_sink_1; log_filter_internal; log_sink_2

In this case, log events pass to the first writer, then to the built-in filter, then to the second writer. The first writer receives unfiltered events. The second writer receives filtered events. You might configure error logging this way if you want one log that contains messages for all log events, and another log that contains messages only for a subset of log events.

5.4.2.2 Default Error Log Destination Configuration

This section discusses which server options configure the default error log destination, which can be the console or a named file. It also indicates which log writer components base their own output destination on the default destination.

In this discussion, console” means stderr, the standard error output. This is your terminal or console window unless the standard error output has been redirected to a different destination.

The server interprets options that determine the default error log destination somewhat differently for Windows and Unix systems. Be sure to configure the destination using the information appropriate to your platform:

After the server interprets the default error log destination options, it sets the log_error system variable to indicate the default destination, which affects where several log writer components write error messages. See How the Default Error Log Destination Affects Log Writers.

Default Error Log Destination on Windows

On Windows, mysqld uses the --log-error, --pid-file, and --console options to determine whether the default error log destination is the console or a file, and, if a file, the file name:

  • If --console is given, the default destination is the console. (--console takes precedence over --log-error if both are given, and the following items regarding --log-error do not apply.)

  • If --log-error is not given, or is given without naming a file, the default destination is a file named host_name.err in the data directory, unless the --pid-file option is specified. In that case, the file name is the PID file base name with a suffix of .err in the data directory.

  • If --log-error is given to name a file, the default destination is that file (with an .err suffix added if the name has no suffix), located under the data directory unless an absolute path name is given to specify a different location.

If the default error log destination is the console, the server sets the log_error system variable to stderr. Otherwise, the default destination is a file and the server sets log_error to the file name.

Default Error Log Destination on Unix and Unix-Like Systems

On Unix and Unix-like systems, mysqld uses the --log-error option to determine whether the default error log destination is the console or a file, and, if a file, the file name:

  • If --log-error is not given, the default destination is the console.

  • If --log-error is given without naming a file, the default destination is a file named host_name.err in the data directory.

  • If --log-error is given to name a file, the default destination is that file (with an .err suffix added if the name has no suffix), located under the data directory unless an absolute path name is given to specify a different location.

  • If --log-error is given in an option file in a [mysqld], [server], or [mysqld_safe] section, mysqld_safe finds and uses the option, and passes it to mysqld.

Note

It is common for Yum or APT package installations to configure an error log file location under /var/log with an option like log-error=/var/log/mysqld.log in a server configuration file. Removing the file name from the option causes the host_name.err file in the data directory to be used.

If the default error log destination is the console, the server sets the log_error system variable to stderr. Otherwise, the default destination is a file and the server sets log_error to the file name.

How the Default Error Log Destination Affects Log Writers

After the server interprets the error log destination configuration options, it sets the log_error system variable to indicate the default error log destination. Log writer components may base their own output destination on the log_error value, or determine their destination independently of log_error

If log_error is stderr, the default error log destination is the console, and log writers that base their output destination on the default destination also write to the console:

  • log_sink_internal, log_sink_json, log_sink_test: These writers write to the console. This is true even for writers such as log_sink_json that can be enabled multiple times; all instances write to the console.

  • log_sink_syseventlog: This writer writes to the system log, regardless of the log_error value.

If log_error is not stderr, the default error log destination is a file and log_error indicates the file name. Log writers that base their output destination on the default destination base output file naming on that file name. (A writer might use exactly that name, or it might use some variant thereof.) Suppose that the log_error value file_name. Then log writers use the name like this:

  • log_sink_internal, log_sink_test: These writers write to file_name.

  • log_sink_json: Successive instances of this writer named in the log_error_services value write to files named file_name plus a numbered .NN.json suffix: file_name.00.json, file_name.01.json, and so forth.

  • log_sink_syseventlog: This writer writes to the system log, regardless of the log_error value.

5.4.2.3 Types of Error Log Filtering

Error log configuration normally includes one log filter component and one or more log writer component. For error log filtering, MySQL offers a choice of components:

5.4.2.4 Priority-Based Error Log Filtering (log_filter_internal)

The log_filter_internal log filter component implements a simple form of log filtering based on error event priority and error code. To affect how log_filter_internal permits or suppresses error, warning, and information events intended for the error log, set the log_error_verbosity and log_error_suppression_list system variables.

log_filter_internal is built in and enabled by default. If this filter is disabled, log_error_verbosity and log_error_suppression_list have no effect, so filtering must be modeled using another filter service instead where desired (for example, with individual filter rules when using log_filter_dragnet). For information about filter configuration, see Section 5.4.2.1, “Error Log Component Configuration”.

Verbosity Filtering

Events intended for the error log have a priority of ERROR, WARNING, or INFORMATION. The log_error_verbosity system variable controls verbosity based on which priorities to permit for messages written to the log, as shown in the following table.

Permitted Message Prioritieslog_error_verbosity Value
ERROR 1
ERROR, WARNING 2
ERROR, WARNING, INFORMATION 3

If log_error_verbosity is 2 or greater, the server logs messages about statements that are unsafe for statement-based logging. If the value is 3, the server logs aborted connections and access-denied errors for new connection attempts. See Section B.4.2.10, “Communication Errors and Aborted Connections”.

If you use replication, a log_error_verbosity value of 2 or greater is recommended, to obtain more information about what is happening, such as messages about network failures and reconnections.

If log_error_verbosity is 2 or greater on a slave server, the slave prints messages to the error log to provide information about its status, such as the binary log and relay log coordinates where it starts its job, when it is switching to another relay log, when it reconnects after a disconnect, and so forth.

There is also a priority of SYSTEM. System messages about non-error situations are printed to the error log regardless of the log_error_verbosity value. These messages include startup and shutdown messages, and some significant changes to settings.

In the MySQL error log, system messages are labeled as System”. Other log writers might or might not follow the same convention, and in the resulting logs, system messages might be assigned the label used for the information priority level, such as Note” or Information”. If you apply any additional filtering or redirection for logging based on the labeling of messages, system messages do not override your filter, but are handled by it in the same way as other messages.

Suppression-List Filtering

The log_error_suppression_list system variable applies to events intended for the error log and specifies which events to suppress when they occur with a priority of WARNING or INFORMATION. For example, if a particular type of warning is considered undesirable noise” in the error log because it occurs frequently but is not of interest, it can be suppressed.

The log_error_suppression_list value may be the empty string for no suppression, or a list of one or more comma-separated values indicating the error codes to suppress. Error codes may be specified in symbolic or numeric form. A numeric code may be specified with or without the MY- prefix. Leading zeros in the numeric part are not significant. Examples of permitted code formats:

ER_SERVER_SHUTDOWN_COMPLETE
MY-000031
000031
MY-31
31

Symbolic values are preferable to numeric values for readability and portability.

Although codes to be suppressed can be expressed in symbolic or numeric form, the numeric value of each code must be in a permitted range:

  • 1 to 999: Global error codes that are used by the server as well as by clients.

  • 10000 and higher: Server error codes intended to be written to the error log (not sent to clients).

In addition, each error code specified must actually be used by MySQL. Attempts to specify a code not within a permitted range or within a permitted range but not used by MySQL produce an error and the log_error_suppression_list value remains unchanged.

For information about error code ranges and the error symbols and numbers defined within each range, see Appendix B, Errors, Error Codes, and Common Problems.

The server can generate messages for a given error code at differing priorities, so suppression of a message associated with an error code listed in log_error_suppression_list depends on its priority. Suppose that the variable has a value of 'ER_PARSER_TRACE,MY-010001,10002'. Then log_error_suppression_list has these effects:

  • Messages for those codes are suppressed if generated with a priority of WARNING or INFORMATION.

  • Messages generated with a priority of ERROR or SYSTEM are not suppressed.

Verbosity and Suppression-List Interaction

The effect of log_error_verbosity combines with that of log_error_suppression_list. Consider a server started with these settings:

[mysqld]
log_error_verbosity=2     # error and warning messages only
log_error_suppression_list='ER_PARSER_TRACE,MY-010001,10002'

In this case, log_error_verbosity permits messages with ERROR or WARNING priority and discards messages with INFORMATION priority. Of the nondiscarded messages, log_error_suppression_list discards messages with WARNING priority and any of the named error codes.

Note

The log_error_verbosity value of 2 shown in the example is also its default value, so the effect of this variable on INFORMATION messages is as just described by default, without an explicit setting. You must set log_error_verbosity to 3 if you want log_error_suppression_list to affect messages with INFORMATION priority.

Consider a server started with this setting:

[mysqld]
log_error_verbosity=1     # error messages only

In this case, log_error_verbosity permits messages with ERROR priority and discards messages with WARNING or INFORMATION priority. Setting log_error_suppression_list would have no effect because all error codes it might suppress are already discarded due to the log_error_verbosity setting.

5.4.2.5 Rule-Based Error Log Filtering (log_filter_dragnet)

The log_filter_dragnet log filter component enables log filtering based on user-defined rules. To define the applicable rules, set the dragnet.log_error_filter_rules system variable.

To enable the log_filter_dragnet filter, first load the filter component, then modify the log_error_services value. The following example enables log_filter_dragnet in combination with the built-in log writer:

INSTALL COMPONENT 'file://component_log_filter_dragnet';
SET GLOBAL log_error_services = 'log_filter_dragnet; log_sink_internal';

To set log_error_services to take effect at server startup, use the instructions at Section 5.4.2.1, “Error Log Component Configuration”. Those instructions apply to other error-logging system variables as well.

With log_filter_dragnet enabled, define its filter rules by setting the dragnet.log_error_filter_rules system variable. A rule set consists of zero or more rules, where each rule is an IF statement terminated by a period (.) character. If the variable value is empty (zero rules), no filtering occurs.

Example 1. This rule set drops information events, and, for other events, removes the source_line field:

SET GLOBAL dragnet.log_error_filter_rules =
  'IF prio>=INFORMATION THEN drop. IF EXISTS source_line THEN unset source_line.';

The effect is similar to the filtering performed by the log_sink_internal filter with a setting of log_error_verbosity=2.

Example 2: This rule limits information events to no more than one per 60 seconds:

SET GLOBAL dragnet.log_error_filter_rules =
  'IF prio>=INFORMATION THEN throttle 1/60.';

Once you have the filtering configuration set up as you desire, consider assigning dragnet.log_error_filter_rules using SET PERSIST rather than SET GLOBAL to make the setting persist across server restarts. Alternatively, add the setting to the server option file.

To stop using the filtering language, first remove it from the set of error logging components. Usually this means using a different filter component rather than no filter component. For example:

SET GLOBAL log_error_services = 'log_filter_internal; log_sink_internal';

Again, consider using using SET PERSIST rather than SET GLOBAL to make the setting persist across server restarts.

Then uninstall the filter log_filter_dragnet component:

UNINSTALL COMPONENT 'file://component_log_filter_dragnet';

The following sections describe aspects of log_filter_dragnet operation in more detail:

log_filter_dragnet Rule Language

The following grammar defines the language for log_filter_dragnet filter rules. Each rule is an IF statement terminated by a period (.) character. The language is not case sensitive.

rule:
    IF condition THEN action
    [ELSEIF condition THEN action] ...
    [ELSE action]
    .

condition: {
    field comparator value
  | [NOT] EXISTS field
  | condition {AND | OR}  condition
}

action: {
    drop
  | throttle {count | count / window_size}
  | set field [:= | =] value
  | unset [field]
}

field: {
    core_field
  | optional_field
  | user_defined_field
}

core_field: {
    time
  | msg
  | prio
  | label
  | err_code
  | err_symbol
  | SQL_state
  | subsystem
}

optional_field: {
    OS_errno
  | OS_errmsg
  | user
  | host
  | thread
  | query_id
  | source_file
  | source_line
  | function
}

user_defined_field:
    sequence of characters in [a-zA-Z0-9_] class

comparator: {== | != | <> | >= | => | <= | =< | < | >}

value: {
    string_literal
  | integer_literal
  | float_literal
  | error_symbol
  | priority
}

count: integer_literal
window_size: integer_literal

string_literal:
    sequence of characters quoted as '...' or "..."

integer_literal:
    sequence of characters in [0-9] class

float_literal:
    integer_literal[.integer_literal]

error_symbol:
    valid MySQL error symbol such as ER_ACCESS_DENIED_ERROR or ER_STARTUP

priority: {
    ERROR
  | WARNING
  | INFORMATION
}

Simple conditions compare a field to a value or test field existence. To construct more complex conditions, use the AND and OR operators. Both operators have the same precedence and evaluate left to right.

To escape a character within a string, precede it by a backslash (\). A backslash is required to include backslash itself or the string-quoting character, optional for other characters.

For convenience, log_filter_dragnet supports symbolic names for comparisons to certain fields. Where applicable, symbolic values are preferable to numeric values for readability and portability.

  • Event priority values 1, 2, and 3 can be specified as ERROR, WARNING, and INFORMATION. Priority symbols are recognized only in comparisons with the prio field. These comparisons are equivalent:

    IF prio == INFORMATION THEN ...
    IF prio == 3 THEN ...
  • Error codes can be specified in numeric form or as the corresponding error symbol. For example, ER_STARTUP is the symbolic name for error 1408, so these comparisons are equivalent:

    IF err_code == ER_STARTUP THEN ...
    IF err_code == 1408 THEN ...

    Error symbols are recognized only in comparisons with the err_code field and user-defined fields.

    To find the error symbol corresponding to a given error code number, use one of these methods:

    Suppose that a rule set with error numbers looks like this:

    IF err_code == 10927 OR err_code == 10914 THEN drop.
    IF err_code == 1131 THEN drop.

    Using perror, determine the error symbols:

    shell> perror 10927 10914 1131
    MySQL error code MY-010927 (ER_ACCESS_DENIED_FOR_USER_ACCOUNT_LOCKED):
    Access denied for user '%-.48s'@'%-.64s'. Account is locked.
    MySQL error code MY-010914 (ER_ABORTING_USER_CONNECTION):
    Aborted connection %u to db: '%-.192s' user: '%-.48s' host:
    '%-.64s' (%-.64s).
    MySQL error code MY-001131 (ER_PASSWORD_ANONYMOUS_USER):
    You are using MySQL as an anonymous user and anonymous users
    are not allowed to change passwords
    

    Substituting error symbols for numbers, the rule set becomes:

    IF err_code == ER_ACCESS_DENIED_FOR_USER_ACCOUNT_LOCKED
      OR err_code == ER_ABORTING_USER_CONNECTION THEN drop.
    IF err_code == ER_PASSWORD_ANONYMOUS_USER THEN drop.

Symbolic names can be specified as quoted strings for comparison with string fields, but in such cases the names are strings that have no special meaning and log_filter_dragnet does not resolve them to the corresponding numeric value. Also, typos may go undetected, whereas an error is thrown immediately on SET for attempts to use an unquoted symbol unknown to the server.

log_filter_dragnet Rule Actions

log_filter_dragnet supports these actions in filter rules:

  • drop: Drop the current log event (do not log it).

  • throttle: Apply rate limiting to reduce log verbosity for events matching particular conditions. The argument indicates a rate, in the form count or count/window_size. The count value indicates the permitted number of events to log per time window. The window_size value is the time window in seconds; if omitted, the default window is 60 seconds. Both values must be integer literals.

    This rule throttles plugin-shutdown messages to 5 per 60 seconds:

    IF err_code == ER_PLUGIN_SHUTTING_DOWN_PLUGIN THEN throttle 5.

    This rule throttles errors and warnings to 1000 per hour and information messages to 100 per hour:

    IF prio <= INFORMATION THEN throttle 1000/3600 ELSE throttle 100/3600.
  • set: Assign a value to a field (and cause the field to exist if it did not already). In subsequent rules, EXISTS tests against the field name are true, and the new value can be tested by comparison conditions.

  • unset: Discard a field. In subsequent rules, EXISTS tests against the field name are false, and comparisons of the field against any value are false.

    In the special case that the condition refers to exactly one field name, the field name following unset is optional and unset discards the named field. These rules are equivalent:

    IF myfield == 2 THEN unset myfield.
    IF myfield == 2 THEN unset.
log_filter_dragnet Rule Fields

log_filter_dragnet supports core, optional, and user-defined fields in rules:

  • A core field is set up automatically for error events. However, its presence in the event is not guaranteed because a core field, like any type of field, may be unset by filter rules. If so, the field will be found missing by later rules within the rule set and by components that execute after the filter (such as log writers).

  • An optional field is normally absent but may be present for certain event types. When present, an optional field provides additional event information as appropriate and available.

  • A user-defined field is any field with a name that is not already defined as a core or optional field. A user-defined field does not exist until created with the set action.

As implied by the preceding description, any given field may be absent, either because it was not present in the first place, or was discarded by a filtering rule. For log writers, the effect of field absence is writer specific. For example, a writer might omit the field from the log message, indicate that the field is missing, or substitute a default. When in doubt, use a filter rule to unset the field, then check what the log writer does with it.

These fields are core fields:

  • time

    The event timestamp.

  • msg

    The event message string.

  • prio

    The event priority, to indicate error, warning, or note/information event. This field corresponds to severity in syslog.

    In comparisons, each priority can be specified as a symbolic priority name or an integer literal. Priority symbols are recognized only in comparisons with the prio field. These comparisons are equivalent:

    IF prio == INFORMATION THEN ...
    IF prio == 3 THEN ...

    The following table shows the permitted priority levels.

    Event TypePriority SymbolNumeric Priority
    Error events ERROR 1
    Warning events WARNING 2
    Note/information events INFORMATION 3

    There is also a priority of SYSTEM, but system messages cannot be filtered and are always written to the error log.

    In general, message priorities are determined as follows:

    Is the situation or event actionable?

    • Yes: Is the situation or event ignorable?

      • Yes: Priority is WARNING.

      • No: Priority is ERROR.

    • No: Is the situation or event mandatory?

      • Yes: Priority is SYSTEM.

      • No: Priority is INFORMATION.

    Priority values follow the principle that higher priorities have lower values, and vice versa. Priority values begin at 1 for the most severe events (errors) and increase for events with decreasing priority. For example, to discard events with lower priority than warnings, test for priority values higher than WARNING:

    IF prio > WARNING THEN drop.

    The following examples show the log_filter_dragnet rules to achieve an effect similar to each log_error_verbosity value permitted by the log_filter_internal filter:

    • Errors only (log_error_verbosity=1):

      IF prio > ERROR THEN drop.
    • Errors and warnings (log_error_verbosity=2):

      IF prio > WARNING THEN drop.
    • Errors, warnings, and notes (log_error_verbosity=3):

      IF prio > INFORMATION THEN drop.

      This rule can actually be omitted because there are no prio values greater than INFORMATION, so effectively it drops nothing.

  • err_code

    The numeric event error code. In comparisons, the value to test can be specified as a symbolic error name or an integer literal. Error symbols are recognized only in comparisons with the err_code field and user-defined fields. These comparisons are equivalent:

    IF err_code == ER_ACCESS_DENIED_ERROR THEN ...
    IF err_code == 1045 THEN ...
  • err_symbol

    The event error symbol, as a string (for example, 'ER_DUP_KEY'). err_symbol values are intended more for identifying particular lines in log output than for use in filter rule comparisons because log_filter_dragnet does not resolve comparison values specified as strings to the equivalent numeric error code.

  • SQL_state

    The event SQLSTATE value, as a string (for example, '23000').

  • subsystem

    The subsystem in which the event occurred. Possible values are InnoDB (the InnoDB storage engine), Repl (the replication subsystem), Server (otherwise).

Optional fields fall into the following categories:

Additional information about the error, such as the error signaled by the operating system or the error lable:

  • OS_errno

    The operating system error number.

  • OS_errmsg

    The operating system error message.

  • label

    The label corresponding to the prio value, as a string. Filter rules can change the label for log writers that support custom labels. label values are intended more for identifying particular lines in log output than for use in filter rule comparisons because log_filter_dragnet does not resolve comparison values specified as strings to the equivalent numeric priority.

Identification of the client for which the event occurred:

  • user

    The client user.

  • host

    The client host.

  • thread

    The thread ID.

  • query_id

    The query ID.

Debugging information:

  • source_file

    The source file in which the event occurred. The file name should omit any leading path. For example, to test for the sql/gis/distance.cc file, write the comparison like this:

    IF source_file == "distance.cc" THEN ...
  • source_line

    The line within the source file at which the event occurred.

  • function

    The function in which the event occurred.

  • component

    The component or plugin in which the event occurred.

5.4.2.6 Error Logging in JSON Format

This section describes how to configure error logging using the built-in filter, log_filter_internal, and the JSON writer, log_sink_json, to take effect immediately and for subsequent server startups. For general information about configuring error logging, see Section 5.4.2.1, “Error Log Component Configuration”.

To enable the JSON writer, first load the writer component, then modify the log_error_services value:

INSTALL COMPONENT 'file://component_log_sink_json';
SET GLOBAL log_error_services = 'log_filter_internal; log_sink_json';

To set log_error_services to take effect at server startup, use the instructions at Section 5.4.2.1, “Error Log Component Configuration”. Those instructions apply to other error-logging system variables as well.

It is permitted to name log_sink_json multiple times in the log_error_services value. For example, to write unfiltered events with one instance and filtered events with another instance, you could set log_error_services like this:

SET GLOBAL log_error_services = 'log_sink_json; log_filter_internal; log_sink_json';

The JSON log writer determines its output destination based on the default error log destination, which is given by the log_error system variable. If log_error names a file, the JSON writer bases output file naming on that file name, plus a numbered .NN.json suffix, with NN starting at 00. For example, if log_error is file_name, successive instances of log_sink_json named in the log_error_services value write to file_name.00.json, file_name.01.json, and so forth.

If log_error is stderr, the JSON writer writes to the console. If log_json_writer is named multiple times in the log_error_services value, they all write to the console, which is likely not useful.

5.4.2.7 Error Logging to the System Log

It is possible to have mysqld write the error log to the system log (the Event Log on Windows, and syslog on Unix and Unix-like systems).

This section describes how to configure error logging using the built-in filter, log_filter_internal, and the system log writer, log_sink_syseventlog, to take effect immediately and for subsequent server startups. For general information about configuring error logging, see Section 5.4.2.1, “Error Log Component Configuration”.

To enable the system log writer, first load the writer component, then modify the log_error_services value:

INSTALL COMPONENT 'file://component_log_sink_syseventlog';
SET GLOBAL log_error_services = 'log_filter_internal; log_sink_syseventlog';

To set log_error_services to take effect at server startup, use the instructions at Section 5.4.2.1, “Error Log Component Configuration”. Those instructions apply to other error-logging system variables as well.

Note

For MySQL 8.0 configuration, you must enable error logging to the system log explicitly. This differs from MySQL 5.7 and earlier, for which error logging to the system log is enabled by default on Windows, and on all platforms requires no component loading.

Error logging to the system log may require additional system configuration. Consult the system log documentation for your platform.

On Windows, error messages written to the Event Log within the Application log have these characteristics:

  • Entries marked as Error, Warning, and Note are written to the Event Log, but not messages such as information statements from individual storage engines.

  • Event Log entries have a source of MySQL (or MySQL-tag if syseventlog.tag is defined as tag).

On Unix and Unix-like systems, logging to the system log uses syslog. The following system variables affect syslog messages:

  • syseventlog.facility: The default facility for syslog messages is daemon. Set this variable to specify a different facility.

  • syseventlog.include_pid: Whether to include the server process ID in each line of syslog output.

  • syseventlog.tag: This variable defines a tag to add to the server identifier (mysqld) in syslog messages. If defined, the tag is appended to the identifier with a leading hyphen.

Note

Prior to MySQL 8.0.13, use the log_syslog_facility, log_syslog_include_pid, and log_syslog_tag system variables rather than the syseventlog.xxx variables.

MySQL uses the custom label System” for important system messages about non-error situations, such as startup, shutdown, and some significant changes to settings. In logs that do not support custom labels, including the Event Log on Windows, and syslog on Unix and Unix-like systems, system messages are assigned the label used for the information priority level. However, these messages are printed to the log even if the MySQL log_error_verbosity setting normally excludes messages at the information level.

When a log writer must fall back to a label of Information” instead of System” in this way, and the log event is further processed outside of the MySQL server (for example, filtered or forwarded by a syslog configuration), these events may by default be processed by the secondary application as being of Information” priority rather than System” priority.

5.4.2.8 Error Log Message Format

Each error log sink (writer) component has a characteristic output format it uses to write messages to its destination, but other factors may influence the content of the messages:

For all log writers, the ID included in error log messages is that of the thread within mysqld responsible for writing the message. This indicates which part of the server produced the message, and is consistent with general query log and slow query log messages, which include the connection thread ID.

Output Format for log_sink_internal

This log writer produces the traditional error log output. It writes messages using this format:

timestamp thread_id [priority] [err_code] [subsystem] message

The [ and ] square bracket characters are literal characters in the message format. They do not indicate that fields are optional.

The [err_code] and [subsystem] fields were added in MySQL 8.0. They will be missing from logs generated by older servers. Log parsers can treat these fields as parts of the message text that will be present only for logs written by servers recent enough to include them. Parsers must treat the err_code part of [err_code] indicators as a string value.

Examples:

2018-03-22T12:35:47.538083Z 0 [Note] [MY-012487] [InnoDB] InnoDB: DDL log recovery : begin
2018-03-22T12:35:47.550565Z 0 [Warning] [MY-010068] [Server] CA certificate /var/mysql/sslinfo/cacert.pem is self signed.
2018-03-22T12:35:47.669397Z 4 [Note] [MY-010051] [Server] Event Scheduler: scheduler thread started with id 4
2018-03-22T12:35:47.550939Z 0 [Note] [MY-010253] [Server] IPv6 is available.
Output Format for log_sink_json

The JSON-format log writer produces messages as JSON objects that contain key-value pairs. For example:

{ "prio": 3, "err_code": 10051, "subsystem": "Server",
  "source_file": "event_scheduler.cc", "function": "run",
  "msg": "Event Scheduler: scheduler thread started with id 4",
  "time": "2018-03-22T12:35:47.669397Z", "thread": 4,
  "err_symbol": "ER_SCHEDULER_STARTED", "SQL_state": "HY000",
"label": "Note" }
Output Format for log_sink_syseventlog

The system log writer produces output that conforms to the system log format used on the local platform.

Output Format for Early-Startup Logging

The server generates some error log messages before startup options have been processed, and thus before it knows error log settings such as the log_error_verbosity and log_timestamps values, and which log components are to be used. The server handles error log messages that are generated early in the startup process as follows:

  • Prior to MySQL 8.0.14, the server generates messages with the default timestamp, format, and verbosity level, and buffers them. After the startup options are processed and the error log configuration is known, the server flushes the buffered messages. Because these early messages use the default log configuration, they may differ from what is specified by the startup options. Also, the early messages are not flushed to log writers other than the default. For example, logging to the JSON writer does not include these early messages because they are not in JSON format.

  • As of MySQL 8.0.14, the server buffers log events rather than formatted log messages. This enables it to retroactively apply configuration settings to those events after the settings are known, with the result that flushed messages use the configured settings, not the defaults. Also, messages are flushed to all configured writers, not just the default writer.

    If a fatal error occurs before log configuration is known and the server must exit, the server so they are not lost. If no fatal error occurs but startup is excessively slow prior to processing startup options, the server periodically formats and flushes buffered messages using the logging defaults so as not to appear unresponsive. Although these behaviors are similar to pre-8.0.14 behavior in that the defaults are used, they are preferable to losing messages when exceptional conditions occur.

System Variables That Affect Error Log Format

The log_timestamps system variable controls the time zone of timestamps in messages written to the error log (as well as to general query log and slow query log files). Permitted values are UTC (the default) and SYSTEM (local system time zone).

5.4.2.9 Error Log File Flushing and Renaming

If you flush the error log using FLUSH ERROR LOGS, FLUSH LOGS, or mysqladmin flush-logs, the server closes and reopens any error log file to which it is writing. To rename an error log file, do so manually before flushing. Flushing the logs then opens a new file with the original file name. For example, assuming a log file name of host_name.err, to rename the file and create a new one, use the following commands:

mv host_name.err host_name.err-old
mysqladmin flush-logs
mv host_name.err-old backup-directory

On Windows, use rename rather than mv.

If the location of the error log file is not writable by the server, the log-flushing operation fails to create a new log file. For example, on Linux, the server might write the error log to the /var/log/mysqld.log file, where the /var/log directory is owned by root and is not writable by mysqld. For information about handling this case, see Section 5.4.7, “Server Log Maintenance”.

If the server is not writing to a named error log file, no error log file renaming occurs when the error log is flushed.

5.4.3 The General Query Log

The general query log is a general record of what mysqld is doing. The server writes information to this log when clients connect or disconnect, and it logs each SQL statement received from clients. The general query log can be very useful when you suspect an error in a client and want to know exactly what the client sent to mysqld.

Each line that shows when a client connects also includes using connection_type to indicate the protocol used to establish the connection. connection_type is one of TCP/IP (TCP/IP connection established without SSL), SSL/TLS (TCP/IP connection established with SSL), Socket (Unix socket file connection), Named Pipe (Windows named pipe connection), or Shared Memory (Windows shared memory connection).

mysqld writes statements to the query log in the order that it receives them, which might differ from the order in which they are executed. This logging order is in contrast with that of the binary log, for which statements are written after they are executed but before any locks are released. In addition, the query log may contain statements that only select data while such statements are never written to the binary log.

When using statement-based binary logging on a replication master server, statements received by its slaves are written to the query log of each slave. Statements are written to the query log of the master server if a client reads events with the mysqlbinlog utility and passes them to the server.

However, when using row-based binary logging, updates are sent as row changes rather than SQL statements, and thus these statements are never written to the query log when binlog_format is ROW. A given update also might not be written to the query log when this variable is set to MIXED, depending on the statement used. See Section 17.2.1.1, “Advantages and Disadvantages of Statement-Based and Row-Based Replication”, for more information.

By default, the general query log is disabled. To specify the initial general query log state explicitly, use --general_log[={0|1}]. With no argument or an argument of 1, --general_log enables the log. With an argument of 0, this option disables the log. To specify a log file name, use --general_log_file=file_name. To specify the log destination, use the log_output system variable (as described in Section 5.4.1, “Selecting General Query Log and Slow Query Log Output Destinations”).

Note

If you specify the TABLE log destination, see Log Tables and Too many open files” Errors.

If you specify no name for the general query log file, the default name is host_name.log. The server creates the file in the data directory unless an absolute path name is given to specify a different directory.

To disable or enable the general query log or change the log file name at runtime, use the global general_log and general_log_file system variables. Set general_log to 0 (or OFF) to disable the log or to 1 (or ON) to enable it. Set general_log_file to specify the name of the log file. If a log file already is open, it is closed and the new file is opened.

When the general query log is enabled, the server writes output to any destinations specified by the log_output system variable. If you enable the log, the server opens the log file and writes startup messages to it. However, further logging of queries to the file does not occur unless the FILE log destination is selected. If the destination is NONE, the server writes no queries even if the general log is enabled. Setting the log file name has no effect on logging if the log destination value does not contain FILE.

Server restarts and log flushing do not cause a new general query log file to be generated (although flushing closes and reopens it). To rename the file and create a new one, use the following commands:

shell> mv host_name.log host_name-old.log
shell> mysqladmin flush-logs
shell> mv host_name-old.log backup-directory

On Windows, use rename rather than mv.

You can also rename the general query log file at runtime by disabling the log:

SET GLOBAL general_log = 'OFF';

With the log disabled, rename the log file externally (for example, from the command line). Then enable the log again:

SET GLOBAL general_log = 'ON';

This method works on any platform and does not require a server restart.

To disable or enable general query logging for the current session, set the session sql_log_off variable to ON or OFF. (This assumes that the general query log itself is enabled.)

Passwords in statements written to the general query log are rewritten by the server not to occur literally in plain text. Password rewriting can be suppressed for the general query log by starting the server with the --log-raw option. This option may be useful for diagnostic purposes, to see the exact text of statements as received by the server, but for security reasons is not recommended for production use. See also Section 6.1.2.3, “Passwords and Logging”.

An implication of password rewriting is that statements that cannot be parsed (due, for example, to syntax errors) are not written to the general query log because they cannot be known to be password free. Use cases that require logging of all statements including those with errors should use the --log-raw option, bearing in mind that this also bypasses password rewriting.

Password rewriting occurs only when plain text passwords are expected. For statements with syntax that expect a password hash value, no rewriting occurs. If a plain text password is supplied erroneously for such syntax, the password is logged as given, without rewriting.

The log_timestamps system variable controls the time zone of timestamps in messages written to the general query log file (as well as to the slow query log file and the error log). It does not affect the time zone of general query log and slow query log messages written to log tables, but rows retrieved from those tables can be converted from the local system time zone to any desired time zone with CONVERT_TZ() or by setting the session time_zone system variable.

5.4.4 The Binary Log

The binary log contains events” that describe database changes such as table creation operations or changes to table data. It also contains events for statements that potentially could have made changes (for example, a DELETE which matched no rows), unless row-based logging is used. The binary log also contains information about how long each statement took that updated data. The binary log has two important purposes:

  • For replication, the binary log on a master replication server provides a record of the data changes to be sent to slave servers. The master server sends the events contained in its binary log to its slaves, which execute those events to make the same data changes that were made on the master. See Section 17.2, “Replication Implementation”.

  • Certain data recovery operations require use of the binary log. After a backup has been restored, the events in the binary log that were recorded after the backup was made are re-executed. These events bring databases up to date from the point of the backup. See Section 7.5, “Point-in-Time (Incremental) Recovery Using the Binary Log”.

The binary log is not used for statements such as SELECT or SHOW that do not modify data. To log all statements (for example, to identify a problem query), use the general query log. See Section 5.4.3, “The General Query Log”.

Running a server with binary logging enabled makes performance slightly slower. However, the benefits of the binary log in enabling you to set up replication and for restore operations generally outweigh this minor performance decrement.

The binary log is resilient to unexpected halts. Only complete events or transactions are logged or read back.

Passwords in statements written to the binary log are rewritten by the server not to occur literally in plain text. See also Section 6.1.2.3, “Passwords and Logging”.

From MySQL 8.0.14, binary log files and relay log files can be encrypted, helping to protect these files and the potentially sensitive data contained in them from being misused by outside attackers, and also from unauthorized viewing by users of the operating system where they are stored. You enable encryption on a MySQL server by setting the binlog_encryption system variable to ON. For more information, see Section 17.3.2, “Encrypting Binary Log Files and Relay Log Files”.

The following discussion describes some of the server options and variables that affect the operation of binary logging. For a complete list, see Section 17.1.6.4, “Binary Logging Options and Variables”.

Binary logging is enabled by default (the log_bin system variable is set to ON). The exception is if you use mysqld to initialize the data directory manually by invoking it with the --initialize or --initialize-insecure option, when binary logging is disabled by default, but can be enabled by specifying the --log-bin option.

To disable binary logging, you can specify the --skip-log-bin or --disable-log-bin option at startup. If either of these options is specified and --log-bin is also specified, the option specified later takes precedence.

The --log-slave-updates and --slave-preserve-commit-order options require binary logging. If you disable binary logging, either omit these options, or specify --log-slave-updates=OFF and --skip-slave-preserve-commit-order. MySQL disables these options by default when --skip-log-bin or --disable-log-bin is specified. If you specify --log-slave-updates or --slave-preserve-commit-order together with --skip-log-bin or --disable-log-bin, a warning or error message is issued.

The --log-bin[=base_name] option is used to specify the base name for binary log files. If you do not supply the --log-bin option, MySQL uses binlog as the default base name for the binary log files. For compatibility with earlier releases, if you supply the --log-bin option with no string or with an empty string, the base name defaults to host_name-bin, using the name of the host machine. It is recommended that you specify a base name, so that if the host name changes, you can easily continue to use the same binary log file names (see Section B.4.7, “Known Issues in MySQL”). If you supply an extension in the log name (for example, --log-bin=base_name.extension), the extension is silently removed and ignored.

mysqld appends a numeric extension to the binary log base name to generate binary log file names. The number increases each time the server creates a new log file, thus creating an ordered series of files. The server creates a new file in the series each time it starts or flushes the logs. The server also creates a new binary log file automatically after the current log's size reaches max_binlog_size. A binary log file may become larger than max_binlog_size if you are using large transactions because a transaction is written to the file in one piece, never split between files.

To keep track of which binary log files have been used, mysqld also creates a binary log index file that contains the names of the binary log files. By default, this has the same base name as the binary log file, with the extension '.index'. You can change the name of the binary log index file with the --log-bin-index[=file_name] option. You should not manually edit this file while mysqld is running; doing so would confuse mysqld.

The term binary log file” generally denotes an individual numbered file containing database events. The term binary log” collectively denotes the set of numbered binary log files plus the index file.

The default location for binary log files and the binary log index file is the data directory. You can use the --log-bin option to specify an alternative location, by adding a leading absolute path name to the base name to specify a different directory. When the server reads an entry from the binary log index file, which tracks the binary log files that have been used, it checks whether the entry contains a relative path. If it does, the relative part of the path is replaced with the absolute path set using the --log-bin option. An absolute path recorded in the binary log index file remains unchanged; in such a case, the index file must be edited manually to enable a new path or paths to be used. The binary log file base name and any specified path are available as the log_bin_basename system variable.

In MySQL 5.7, a server ID had to be specified when binary logging was enabled, or the server would not start. In MySQL 8.0, the server_id system variable is set to 1 by default. The server can be started with this default ID when binary logging is enabled, but an informational message is issued if you do not specify a server ID explicitly using the server_id system variable. For servers that are used in a replication topology, you must specify a unique nonzero server ID for each server.

A client that has privileges sufficient to set restricted session system variables (see Section 5.1.9.1, “System Variable Privileges”) can disable binary logging of its own statements by using a SET sql_log_bin=OFF statement.

By default, the server logs the length of the event as well as the event itself and uses this to verify that the event was written correctly. You can also cause the server to write checksums for the events by setting the binlog_checksum system variable. When reading back from the binary log, the master uses the event length by default, but can be made to use checksums if available by enabling the master_verify_checksum system variable. The slave I/O thread also verifies events received from the master. You can cause the slave SQL thread to use checksums if available when reading from the relay log by enabling the slave_sql_verify_checksum system variable.

The format of the events recorded in the binary log is dependent on the binary logging format. Three format types are supported: row-based logging, statement-based logging and mixed-base logging. The binary logging format used depends on the MySQL version. For general descriptions of the logging formats, see Section 5.4.4.1, “Binary Logging Formats”. For detailed information about the format of the binary log, see MySQL Internals: The Binary Log.

The server evaluates the --binlog-do-db and --binlog-ignore-db options in the same way as it does the --replicate-do-db and --replicate-ignore-db options. For information about how this is done, see Section 17.2.5.1, “Evaluation of Database-Level Replication and Binary Logging Options”.

A replication slave server is started with the log_slave_updates system variable enabled by default, meaning that the slave writes to its own binary log any data modifications that are received from the replication master. The binary log must be enabled for this setting to work (see Section 17.1.6.3, “Replication Slave Options and Variables”). This setting enables the slave to act as a master to other slaves in chained replication.

You can delete all binary log files with the RESET MASTER statement, or a subset of them with PURGE BINARY LOGS. See Section 13.7.8.6, “RESET Statement”, and Section 13.4.1.1, “PURGE BINARY LOGS Statement”.

If you are using replication, you should not delete old binary log files on the master until you are sure that no slave still needs to use them. For example, if your slaves never run more than three days behind, once a day you can execute mysqladmin flush-logs on the master and then remove any logs that are more than three days old. You can remove the files manually, but it is preferable to use PURGE BINARY LOGS, which also safely updates the binary log index file for you (and which can take a date argument). See Section 13.4.1.1, “PURGE BINARY LOGS Statement”.

You can display the contents of binary log files with the mysqlbinlog utility. This can be useful when you want to reprocess statements in the log for a recovery operation. For example, you can update a MySQL server from the binary log as follows:

shell> mysqlbinlog log_file | mysql -h server_name

mysqlbinlog also can be used to display replication slave relay log file contents because they are written using the same format as binary log files. For more information on the mysqlbinlog utility and how to use it, see Section 4.6.8, “mysqlbinlog — Utility for Processing Binary Log Files”. For more information about the binary log and recovery operations, see Section 7.5, “Point-in-Time (Incremental) Recovery Using the Binary Log”.

Binary logging is done immediately after a statement or transaction completes but before any locks are released or any commit is done. This ensures that the log is logged in commit order.

Updates to nontransactional tables are stored in the binary log immediately after execution.

Within an uncommitted transaction, all updates (UPDATE, DELETE, or INSERT) that change transactional tables such as InnoDB tables are cached until a COMMIT statement is received by the server. At that point, mysqld writes the entire transaction to the binary log before the COMMIT is executed.

Modifications to nontransactional tables cannot be rolled back. If a transaction that is rolled back includes modifications to nontransactional tables, the entire transaction is logged with a ROLLBACK statement at the end to ensure that the modifications to those tables are replicated.

When a thread that handles the transaction starts, it allocates a buffer of binlog_cache_size to buffer statements. If a statement is bigger than this, the thread opens a temporary file to store the transaction. The temporary file is deleted when the thread ends. From MySQL 8.0.17, if binary log encryption is active on the server, the temporary file is encrypted.

The Binlog_cache_use status variable shows the number of transactions that used this buffer (and possibly a temporary file) for storing statements. The Binlog_cache_disk_use status variable shows how many of those transactions actually had to use a temporary file. These two variables can be used for tuning binlog_cache_size to a large enough value that avoids the use of temporary files.

The max_binlog_cache_size system variable (default 4GB, which is also the maximum) can be used to restrict the total size used to cache a multiple-statement transaction. If a transaction is larger than this many bytes, it fails and rolls back. The minimum value is 4096.

If you are using the binary log and row based logging, concurrent inserts are converted to normal inserts for CREATE ... SELECT or INSERT ... SELECT statements. This is done to ensure that you can re-create an exact copy of your tables by applying the log during a backup operation. If you are using statement-based logging, the original statement is written to the log.

The binary log format has some known limitations that can affect recovery from backups. See Section 17.5.1, “Replication Features and Issues”.

Binary logging for stored programs is done as described in Section 24.7, “Stored Program Binary Logging”.

Note that the binary log format differs in MySQL 8.0 from previous versions of MySQL, due to enhancements in replication. See Section 17.5.2, “Replication Compatibility Between MySQL Versions”.

If the server is unable to write to the binary log, flush binary log files, or synchronize the binary log to disk, the binary log on the replication master can become inconsistent and replication slaves can lose synchronization with the master. The binlog_error_action system variable controls the action taken if an error of this type is encountered with the binary log.

  • The default setting, ABORT_SERVER, makes the server halt binary logging and shut down. At this point, you can identify and correct the cause of the error. On restart, recovery proceeds as in the case of an unexpected server halt (see Section 17.4.2, “Handling an Unexpected Halt of a Replication Slave”).

  • The setting IGNORE_ERROR provides backward compatibility with older versions of MySQL. With this setting, the server continues the ongoing transaction and logs the error, then halts binary logging, but continues to perform updates. At this point, you can identify and correct the cause of the error. To resume binary logging, log_bin must be enabled again, which requires a server restart. Only use this option if you require backward compatibility, and the binary log is non-essential on this MySQL server instance. For example, you might use the binary log only for intermittent auditing or debugging of the server, and not use it for replication from the server or rely on it for point-in-time restore operations.

By default, the binary log is synchronized to disk at each write (sync_binlog=1). If sync_binlog was not enabled, and the operating system or machine (not only the MySQL server) crashed, there is a chance that the last statements of the binary log could be lost. To prevent this, enable the sync_binlog system variable to synchronize the binary log to disk after every N commit groups. See Section 5.1.8, “Server System Variables”. The safest value for sync_binlog is 1 (the default), but this is also the slowest.

In earlier MySQL releases, there was a chance of inconsistency between the table content and binary log content if a crash occurred, even with sync_binlog set to 1. For example, if you are using InnoDB tables and the MySQL server processes a COMMIT statement, it writes many prepared transactions to the binary log in sequence, synchronizes the binary log, and then commits the transaction into InnoDB. If the server crashed between those two operations, the transaction would be rolled back by InnoDB at restart but still exist in the binary log. Such an issue was resolved in previous releases by enabling InnoDB support for two-phase commit in XA transactions. In 8.0.0 and higher, the InnoDB support for two-phase commit in XA transactions is always enabled.

InnoDB support for two-phase commit in XA transactions ensures that the binary log and InnoDB data files are synchronized. However, the MySQL server should also be configured to synchronize the binary log and the InnoDB logs to disk before committing the transaction. The InnoDB logs are synchronized by default, and sync_binlog=1 ensures the binary log is synchronized. The effect of implicit InnoDB support for two-phase commit in XA transactions and sync_binlog=1 is that at restart after a crash, after doing a rollback of transactions, the MySQL server scans the latest binary log file to collect transaction xid values and calculate the last valid position in the binary log file. The MySQL server then tells InnoDB to complete any prepared transactions that were successfully written to the to the binary log, and truncates the binary log to the last valid position. This ensures that the binary log reflects the exact data of InnoDB tables, and therefore the slave remains in synchrony with the master because it does not receive a statement which has been rolled back.

If the MySQL server discovers at crash recovery that the binary log is shorter than it should have been, it lacks at least one successfully committed InnoDB transaction. This should not happen if sync_binlog=1 and the disk/file system do an actual sync when they are requested to (some do not), so the server prints an error message The binary log file_name is shorter than its expected size. In this case, this binary log is not correct and replication should be restarted from a fresh snapshot of the master's data.

The session values of the following system variables are written to the binary log and honored by the replication slave when parsing the binary log:

5.4.4.1 Binary Logging Formats

The server uses several logging formats to record information in the binary log:

  • Replication capabilities in MySQL originally were based on propagation of SQL statements from master to slave. This is called statement-based logging. You can cause this format to be used by starting the server with --binlog-format=STATEMENT.

  • In row-based logging (the default), the master writes events to the binary log that indicate how individual table rows are affected. You can cause the server to use row-based logging by starting it with --binlog-format=ROW.

  • A third option is also available: mixed logging. With mixed logging, statement-based logging is used by default, but the logging mode switches automatically to row-based in certain cases as described below. You can cause MySQL to use mixed logging explicitly by starting mysqld with the option --binlog-format=MIXED.

The logging format can also be set or limited by the storage engine being used. This helps to eliminate issues when replicating certain statements between a master and slave which are using different storage engines.

With statement-based replication, there may be issues with replicating nondeterministic statements. In deciding whether or not a given statement is safe for statement-based replication, MySQL determines whether it can guarantee that the statement can be replicated using statement-based logging. If MySQL cannot make this guarantee, it marks the statement as potentially unreliable and issues the warning, Statement may not be safe to log in statement format.

You can avoid these issues by using MySQL's row-based replication instead.

5.4.4.2 Setting The Binary Log Format

You can select the binary logging format explicitly by starting the MySQL server with --binlog-format=type. The supported values for type are:

  • STATEMENT causes logging to be statement based.

  • ROW causes logging to be row based. This is the default.

  • MIXED causes logging to use mixed format.

The logging format also can be switched at runtime, although note that there are a number of situations in which you cannot do this, as discussed later in this section. Set the global value of the binlog_format system variable to specify the format for clients that connect subsequent to the change:

mysql> SET GLOBAL binlog_format = 'STATEMENT';
mysql> SET GLOBAL binlog_format = 'ROW';
mysql> SET GLOBAL binlog_format = 'MIXED';

An individual client can control the logging format for its own statements by setting the session value of binlog_format:

mysql> SET SESSION binlog_format = 'STATEMENT';
mysql> SET SESSION binlog_format = 'ROW';
mysql> SET SESSION binlog_format = 'MIXED';

Changing the global binlog_format value requires privileges sufficient to set global system variables. Changing the session binlog_format value requires privileges sufficient to set restricted session system variables. See Section 5.1.9.1, “System Variable Privileges”.

There are several reasons why a client might want to set binary logging on a per-session basis:

  • A session that makes many small changes to the database might want to use row-based logging.

  • A session that performs updates that match many rows in the WHERE clause might want to use statement-based logging because it will be more efficient to log a few statements than many rows.

  • Some statements require a lot of execution time on the master, but result in just a few rows being modified. It might therefore be beneficial to replicate them using row-based logging.

There are exceptions when you cannot switch the replication format at runtime:

  • The replication format cannot be changed from within a stored function or a trigger.

  • If the NDB storage engine is enabled.

  • If a session has open temporary tables, the replication format cannot be changed for the session (SET @@SESSION.binlog_format).

  • If any replication channel has open temporary tables, the replication format cannot be changed globally (SET @@GLOBAL.binlog_format or SET @@PERSIST.binlog_format).

  • If any replication channel applier thread is currently running, the replication format cannot be changed globally (SET @@GLOBAL.binlog_format or SET @@PERSIST.binlog_format).

Trying to switch the replication format in any of these cases (or attempting to set the current replication format) results in an error. You can, however, use PERSIST_ONLY (SET @@PERSIST_ONLY.binlog_format) to change the replication format at any time, because this action does not modify the runtime global system variable value, and takes effect only after a server restart.

Switching the replication format at runtime is not recommended when any temporary tables exist, because temporary tables are logged only when using statement-based replication, whereas with row-based replication and mixed replication, they are not logged.

Switching the replication format while replication is ongoing can also cause issues. Each MySQL Server can set its own and only its own binary logging format (true whether binlog_format is set with global or session scope). This means that changing the logging format on a replication master does not cause a slave to change its logging format to match. When using STATEMENT mode, the binlog_format system variable is not replicated. When using MIXED or ROW logging mode, it is replicated but is ignored by the slave.

A replication slave is not able to convert binary log entries received in ROW logging format to STATEMENT format for use in its own binary log. The slave must therefore use ROW or MIXED format if the master does. Changing the binary logging format on the master from STATEMENT to ROW or MIXED while replication is ongoing to a slave with STATEMENT format can cause replication to fail with errors such as Error executing row event: 'Cannot execute statement: impossible to write to binary log since statement is in row format and BINLOG_FORMAT = STATEMENT.' Changing the binary logging format on the slave to STATEMENT format when the master is still using MIXED or ROW format also causes the same type of replication failure. To change the format safely, you must stop replication and ensure that the same change is made on both the master and the slave.

If you are using InnoDB tables and the transaction isolation level is READ COMMITTED or READ UNCOMMITTED, only row-based logging can be used. It is possible to change the logging format to STATEMENT, but doing so at runtime leads very rapidly to errors because InnoDB can no longer perform inserts.

With the binary log format set to ROW, many changes are written to the binary log using the row-based format. Some changes, however, still use the statement-based format. Examples include all DDL (data definition language) statements such as CREATE TABLE, ALTER TABLE, or DROP TABLE.

When row-based binary logging is used, the binlog_row_event_max_size system variable and its corresponding startup option --binlog-row-event-max-size set a soft limit on the maximum size of row events. The default value is 8192 bytes, and the value can only be changed at server startup. Where possible, rows stored in the binary log are grouped into events with a size not exceeding the value of this setting. If an event cannot be split, the maximum size can be exceeded.

The --binlog-row-event-max-size option is available for servers that are capable of row-based replication. Rows are stored into the binary log in chunks having a size in bytes not exceeding the value of this option. The value must be a multiple of 256. The default value is 8192.

Warning

When using statement-based logging for replication, it is possible for the data on the master and slave to become different if a statement is designed in such a way that the data modification is nondeterministic; that is, it is left to the will of the query optimizer. In general, this is not a good practice even outside of replication. For a detailed explanation of this issue, see Section B.4.7, “Known Issues in MySQL”.

For information about logs kept by replication slaves, see Section 17.2.4, “Replication Relay and Status Logs”.

5.4.4.3 Mixed Binary Logging Format

When running in MIXED logging format, the server automatically switches from statement-based to row-based logging under the following conditions:

In earlier releases, when mixed binary logging format was in use, if a statement was logged by row and the session that executed the statement had any temporary tables, all subsequent statements were treated as unsafe and logged in row-based format until all temporary tables in use by that session were dropped. As of MySQL 8.0, operations on temporary tables are not logged in mixed binary logging format, and the presence of temporary tables in the session has no impact on the logging mode used for each statement.

Note

A warning is generated if you try to execute a statement using statement-based logging that should be written using row-based logging. The warning is shown both in the client (in the output of SHOW WARNINGS) and through the mysqld error log. A warning is added to the SHOW WARNINGS table each time such a statement is executed. However, only the first statement that generated the warning for each client session is written to the error log to prevent flooding the log.

In addition to the decisions above, individual engines can also determine the logging format used when information in a table is updated. The logging capabilities of an individual engine can be defined as follows:

  • If an engine supports row-based logging, the engine is said to be row-logging capable.

  • If an engine supports statement-based logging, the engine is said to be statement-logging capable.

A given storage engine can support either or both logging formats. The following table lists the formats supported by each engine.

Storage EngineRow Logging SupportedStatement Logging Supported
ARCHIVE Yes Yes
BLACKHOLE Yes Yes
CSV Yes Yes
EXAMPLE Yes No
FEDERATED Yes Yes
HEAP Yes Yes
InnoDB Yes Yes when the transaction isolation level is REPEATABLE READ or SERIALIZABLE; No otherwise.
MyISAM Yes Yes
MERGE Yes Yes
NDB Yes No

Whether a statement is to be logged and the logging mode to be used is determined according to the type of statement (safe, unsafe, or binary injected), the binary logging format (STATEMENT, ROW, or MIXED), and the logging capabilities of the storage engine (statement capable, row capable, both, or neither). (Binary injection refers to logging a change that must be logged using ROW format.)

Statements may be logged with or without a warning; failed statements are not logged, but generate errors in the log. This is shown in the following decision table. Type, binlog_format, SLC, and RLC columns outline the conditions, and Error / Warning and Logged as columns represent the corresponding actions. SLC stands for statement-logging capable”, and RLC stands for row-logging capable”.

Typebinlog_formatSLCRLCError / WarningLogged as
* * No No Error: Cannot execute statement: Binary logging is impossible since at least one engine is involved that is both row-incapable and statement-incapable. -
Safe STATEMENT Yes No - STATEMENT
Safe MIXED Yes No - STATEMENT
Safe ROW Yes No Error: Cannot execute statement: Binary logging is impossible since BINLOG_FORMAT = ROW and at least one table uses a storage engine that is not capable of row-based logging. -
Unsafe STATEMENT Yes No Warning: Unsafe statement binlogged in statement format, since BINLOG_FORMAT = STATEMENT STATEMENT
Unsafe MIXED Yes No Error: Cannot execute statement: Binary logging of an unsafe statement is impossible when the storage engine is limited to statement-based logging, even if BINLOG_FORMAT = MIXED. -
Unsafe ROW Yes No Error: Cannot execute statement: Binary logging is impossible since BINLOG_FORMAT = ROW and at least one table uses a storage engine that is not capable of row-based logging. -
Row Injection STATEMENT Yes No Error: Cannot execute row injection: Binary logging is not possible since at least one table uses a storage engine that is not capable of row-based logging. -
Row Injection MIXED Yes No Error: Cannot execute row injection: Binary logging is not possible since at least one table uses a storage engine that is not capable of row-based logging. -
Row Injection ROW Yes No Error: Cannot execute row injection: Binary logging is not possible since at least one table uses a storage engine that is not capable of row-based logging. -
Safe STATEMENT No Yes Error: Cannot execute statement: Binary logging is impossible since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine that is not capable of statement-based logging. -
Safe MIXED No Yes - ROW
Safe ROW No Yes - ROW
Unsafe STATEMENT No Yes Error: Cannot execute statement: Binary logging is impossible since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine that is not capable of statement-based logging. -
Unsafe MIXED No Yes - ROW
Unsafe ROW No Yes - ROW
Row Injection STATEMENT No Yes Error: Cannot execute row injection: Binary logging is not possible since BINLOG_FORMAT = STATEMENT. -
Row Injection MIXED No Yes - ROW
Row Injection ROW No Yes - ROW
Safe STATEMENT Yes Yes - STATEMENT
Safe MIXED Yes Yes - STATEMENT
Safe ROW Yes Yes - ROW
Unsafe STATEMENT Yes Yes Warning: Unsafe statement binlogged in statement format since BINLOG_FORMAT = STATEMENT. STATEMENT
Unsafe MIXED Yes Yes - ROW
Unsafe ROW Yes Yes - ROW
Row Injection STATEMENT Yes Yes Error: Cannot execute row injection: Binary logging is not possible because BINLOG_FORMAT = STATEMENT. -
Row Injection MIXED Yes Yes - ROW
Row Injection ROW Yes Yes - ROW

When a warning is produced by the determination, a standard MySQL warning is produced (and is available using SHOW WARNINGS). The information is also written to the mysqld error log. Only one error for each error instance per client connection is logged to prevent flooding the log. The log message includes the SQL statement that was attempted.

If a slave server has log_error_verbosity set to display warnings, the slave prints messages to the error log to provide information about its status, such as the binary log and relay log coordinates where it starts its job, when it is switching to another relay log, when it reconnects after a disconnect, statements that are unsafe for statement-based logging, and so forth.

5.4.4.4 Logging Format for Changes to mysql Database Tables

The contents of the grant tables in the mysql database can be modified directly (for example, with INSERT or DELETE) or indirectly (for example, with GRANT or CREATE USER). Statements that affect mysql database tables are written to the binary log using the following rules:

CREATE TABLE ... SELECT is a combination of data definition and data manipulation. The CREATE TABLE part is logged using statement format and the SELECT part is logged according to the value of binlog_format.

5.4.5 The Slow Query Log

The slow query log consists of SQL statements that take more than long_query_time seconds to execute and require at least min_examined_row_limit rows to be examined. The slow query log can be used to find queries that take a long time to execute and are therefore candidates for optimization. However, examining a long slow query log can be a time-consuming task. To make this easier, you can use the mysqldumpslow command to process a slow query log file and summarize its contents. See Section 4.6.9, “mysqldumpslow — Summarize Slow Query Log Files”.

The time to acquire the initial locks is not counted as execution time. mysqld writes a statement to the slow query log after it has been executed and after all locks have been released, so log order might differ from execution order.

Slow Query Log Parameters

The minimum and default values of long_query_time are 0 and 10, respectively. The value can be specified to a resolution of microseconds.

By default, administrative statements are not logged, nor are queries that do not use indexes for lookups. This behavior can be changed using log_slow_admin_statements and log_queries_not_using_indexes, as described later.

By default, the slow query log is disabled. To specify the initial slow query log state explicitly, use --slow_query_log[={0|1}]. With no argument or an argument of 1, --slow_query_log enables the log. With an argument of 0, this option disables the log. To specify a log file name, use --slow_query_log_file=file_name. To specify the log destination, use the log_output system variable (as described in Section 5.4.1, “Selecting General Query Log and Slow Query Log Output Destinations”).

Note

If you specify the TABLE log destination, see Log Tables and Too many open files” Errors.

If you specify no name for the slow query log file, the default name is host_name-slow.log. The server creates the file in the data directory unless an absolute path name is given to specify a different directory.

To disable or enable the slow query log or change the log file name at runtime, use the global slow_query_log and slow_query_log_file system variables. Set slow_query_log to 0 to disable the log or to 1 to enable it. Set slow_query_log_file to specify the name of the log file. If a log file already is open, it is closed and the new file is opened.

The server writes less information to the slow query log if you use the --log-short-format option.

To include slow administrative statements in the slow query log, enable the log_slow_admin_statements system variable. Administrative statements include ALTER TABLE, ANALYZE TABLE, CHECK TABLE, CREATE INDEX, DROP INDEX, OPTIMIZE TABLE, and REPAIR TABLE.

To include queries that do not use indexes for row lookups in the statements written to the slow query log, enable the log_queries_not_using_indexes system variable. (Even with that variable enabled, the server does not log queries that would not benefit from the presence of an index due to the table having fewer than two rows.)

When queries that do not use an index are logged, the slow query log may grow quickly. It is possible to put a rate limit on these queries by setting the log_throttle_queries_not_using_indexes system variable. By default, this variable is 0, which means there is no limit. Positive values impose a per-minute limit on logging of queries that do not use indexes. The first such query opens a 60-second window within which the server logs queries up to the given limit, then suppresses additional queries. If there are suppressed queries when the window ends, the server logs a summary that indicates how many there were and the aggregate time spent in them. The next 60-second window begins when the server logs the next query that does not use indexes.

The server uses the controlling parameters in the following order to determine whether to write a query to the slow query log:

  1. The query must either not be an administrative statement, or log_slow_admin_statements must be enabled.

  2. The query must have taken at least long_query_time seconds, or log_queries_not_using_indexes must be enabled and the query used no indexes for row lookups.

  3. The query must have examined at least min_examined_row_limit rows.

  4. The query must not be suppressed according to the log_throttle_queries_not_using_indexes setting.

The log_timestamps system variable controls the time zone of timestamps in messages written to the slow query log file (as well as to the general query log file and the error log). It does not affect the time zone of general query log and slow query log messages written to log tables, but rows retrieved from those tables can be converted from the local system time zone to any desired time zone with CONVERT_TZ() or by setting the session time_zone system variable.

By default, a replication slave does not write replicated queries to the slow query log. To change this, enable the log_slow_slave_statements system variable. Note that if row-based replication is in use (binlog_format=ROW), log_slow_slave_statements has no effect. Queries are only added to the slave's slow query log when they are logged in statement format in the binary log, that is, when binlog_format=STATEMENT is set, or when binlog_format=MIXED is set and the statement is logged in statement format. Slow queries that are logged in row format when binlog_format=MIXED is set, or that are logged when binlog_format=ROW is set, are not added to the slave's slow query log, even if log_slow_slave_statements is enabled.

Slow Query Log Contents

When the slow query log is enabled, the server writes output to any destinations specified by the log_output system variable. If you enable the log, the server opens the log file and writes startup messages to it. However, further logging of queries to the file does not occur unless the FILE log destination is selected. If the destination is NONE, the server writes no queries even if the slow query log is enabled. Setting the log file name has no effect on logging if FILE is not selected as an output destination.

If the slow query log is enabled and FILE is selected as an output destination, each statement written to the log is preceded by a line that begins with a # character and has these fields (with all fields on a single line):

  • Query_time: duration

    The statement execution time in seconds.

  • Lock_time: duration

    The time to acquire locks in seconds.

  • Rows_sent: N

    The number of rows sent to the client.

  • Rows_examined:

    The number of rows examined by the server layer (not counting any processing internal to storage engines).

Enabling the log_slow_extra system variable (available as of MySQL 8.0.14) causes the server to write the following extra fields to FILE output in addition to those just listed (TABLE output is unaffected). Some field descriptions refer to status variable names. Consult the status variable descriptions for more information. However, in the slow query log, the counters are per-statement values, not cumulative per-session values.

  • Thread_id: ID

    The statement thread identifier.

  • Errno: error_number

    The statement error number, or 0 if no error occurred.

  • Killed: N

    If the statement was terminated, the error number indicating why, or 0 if the statement terminated normally.

  • Bytes_received: N

    The Bytes_received value for the statement.

  • Bytes_sent: N

    The Bytes_sent value for the statement.

  • Read_first: N

    The Handler_read_first value for the statement.

  • Read_last: N

    The Handler_read_last value for the statement.

  • Read_key: N

    The Handler_read_key value for the statement.

  • Read_next: N

    The Handler_read_next value for the statement.

  • Read_prev: N

    The Handler_read_prev value for the statement.

  • Read_rnd: N

    The Handler_read_rnd value for the statement.

  • Read_rnd_next: N

    The Handler_read_rnd_next value for the statement.

  • Sort_merge_passes: N

    The Sort_merge_passes value for the statement.

  • Sort_range_count: N

    The Sort_range value for the statement.

  • Sort_rows: N

    The Sort_rows value for the statement.

  • Sort_scan_count: N

    The Sort_scan value for the statement.

  • Created_tmp_disk_tables: N

    The Created_tmp_disk_tables value for the statement.

  • Created_tmp_tables: N

    The Created_tmp_tables value for the statement.

  • Start: timestamp

    The statement execution start time.

  • End: timestamp

    The statement execution end time.

A given slow query log file may contain a mix of lines with and without the extra fields added by enabling log_slow_extra. Log file analyzers can determine whether a line contains the additional fields by the field count.

Each statement written to the slow query log file is preceded by a SET statement that includes a timestamp. As of MySQL 8.0.14, the timestamp indicates when the slow statement began executing. Prior to 8.0.14, the timestamp indicates when the slow statement was logged (which occurs after the statement finishes executing).

Passwords in statements written to the slow query log are rewritten by the server not to occur literally in plain text. See Section 6.1.2.3, “Passwords and Logging”.

5.4.6 The DDL Log

The DDL log, or metadata log, records metadata operations generated by data definition statements such as DROP TABLE and ALTER TABLE. MySQL uses this log to recover from crashes occurring in the middle of a metadata operation. When executing the statement DROP TABLE t1, t2, we need to ensure that both t1 and t2 are dropped, and that each table drop is complete. Another example of this type of SQL statement is ALTER TABLE t3 DROP PARTITION p2, where we must make certain that the partition is completely dropped and that its definition is removed from the list of partitions for table t3.

A record of metadata operations such as those just described are written to the file ddl_log.log, in the MySQL data directory. This is a binary file; it is not intended to be human-readable, and you should not attempt to modify its contents in any way.

ddl_log.log is not created until it is actually needed for recording metadata statements, and is removed following a successful start of mysqld. Thus, it is possible for this file not to be present on a MySQL server that is functioning in a completely normal manner.

Currently, ddl_log.log can hold up to 1048573 entries, equivalent to 4 GB in size. Once this limit is exceeded, you must rename or remove the file before it is possible to execute any additional DDL statements. This is a known issue which we are working to resolve (Bug #83708).

There are no user-configurable server options or variables associated with this file.

5.4.7 Server Log Maintenance

As described in Section 5.4, “MySQL Server Logs”, MySQL Server can create several different log files to help you see what activity is taking place. However, you must clean up these files regularly to ensure that the logs do not take up too much disk space.

When using MySQL with logging enabled, you may want to back up and remove old log files from time to time and tell MySQL to start logging to new files. See Section 7.2, “Database Backup Methods”.

On a Linux (Red Hat) installation, you can use the mysql-log-rotate script for log maintenance. If you installed MySQL from an RPM distribution, this script should have been installed automatically. Be careful with this script if you are using the binary log for replication. You should not remove binary logs until you are certain that their contents have been processed by all slaves.

On other systems, you must install a short script yourself that you start from cron (or its equivalent) for handling log files.

Binary log files are automatically removed after the server's binary log expiration period. Removal of the files can take place at startup and when the binary log is flushed. The default binary log expiration period is 30 days. To specify an alternative expiration period, use the binlog_expire_logs_seconds system variable. If you are using replication, you should specify an expiration period that is no lower than the maximum amount of time your slaves might lag behind the master. To remove binary logs on demand, use the PURGE BINARY LOGS statement (see Section 13.4.1.1, “PURGE BINARY LOGS Statement”).

To force MySQL to start using new log files, flush the logs. Log flushing occurs when you execute a FLUSH LOGS statement or a mysqladmin flush-logs, mysqladmin refresh, mysqldump --flush-logs, or mysqldump --master-data command. See Section 13.7.8.3, “FLUSH Statement”, Section 4.5.2, “mysqladmin — Client for Administering a MySQL Server”, and Section 4.5.4, “mysqldump — A Database Backup Program”. In addition, the server flushes the binary log automatically when current binary log file size reaches the value of the max_binlog_size system variable.

FLUSH LOGS supports optional modifiers to enable selective flushing of individual logs (for example, FLUSH BINARY LOGS). See Section 13.7.8.3, “FLUSH Statement”.

A log-flushing operation has the following effects:

  • If binary logging is enabled, the server closes the current binary log file and opens a new log file with the next sequence number.

  • If general query logging or slow query logging to a log file is enabled, the server closes and reopens the log file.

  • If the server was started with the --log-error option to cause the error log to be written to a file, the server closes and reopens the log file.

Execution of log-flushing statements or commands requires connecting to the server using an account that has the RELOAD privilege. On Unix and Unix-like systems, another way to flush the logs is to send a signal to the server, which can be done by root or the account that owns the server process. (See Section 4.10, “Unix Signal Handling in MySQL”.) Signals enable log flushing to be performed without having to connect to the server:

  • A SIGHUP signal flushes all the logs. However, SIGHUP has additional effects other than log flushing that might be undesirable.

  • As of MySQL 8.0.19, SIGUSR1 causes the server to flush the error log, general query log, and slow query log. If you are interested in flushing only those logs, SIGUSR1 can be used as a more lightweight” signal that does not have the SIGHUP effects that are unrelated to logs.

As mentioned previously, flushing the binary log creates a new binary log file, whereas flushing the general query log, slow query log, or error log just closes and reopens the log file. For the latter logs, to cause a new log file to be created on Unix, rename the current log file first before flushing it. At flush time, the server opens the new log file with the original name. For example, if the general query log, slow query log, and error log files are named mysql.log, mysql-slow.log, and err.log, you can use a series of commands like this from the command line:

cd mysql-data-directory
mv mysql.log mysql.log.old
mv mysql-slow.log mysql-slow.log.old
mv err.log err.log.old
mysqladmin flush-logs

On Windows, use rename rather than mv.

At this point, you can make a backup of mysql.log.old, mysql-slow.log.old, and err.log.old, then remove them from disk.

To rename the general query log or slow query log at runtime, first connect to the server and disable the log:

SET GLOBAL general_log = 'OFF';
SET GLOBAL slow_query_log = 'OFF';

With the logs disabled, rename the log files externally (for example, from the command line). Then enable the logs again:

SET GLOBAL general_log = 'ON';
SET GLOBAL slow_query_log = 'ON';

This method works on any platform and does not require a server restart.

Note

For the server to recreate a given log file after you have renamed the file externally, the file location must be writable by the server. This may not always be the case. For example, on Linux, the server might write the error log as /var/log/mysqld.log, where /var/log is owned by root and not writable by mysqld. In this case, log-flushing operations fail to create a new log file.

To handle this situation, you must manually create the new log file with the proper ownership after renaming the original log file. For example, execute these commands as root:

mv /var/log/mysqld.log /var/log/mysqld.log.old
install -omysql -gmysql -m0644 /dev/null /var/log/mysqld.log