linux-system-roles/logging
Table of Contents
Background
Logging role is an abstract layer for provisioning and configuring the logging system. Currently, rsyslog is the only supported provider.
In the nature of logging, there are multiple ways to read logs and multiple ways to output them. For instance, the logging system may read logs from local files, or read them from systemd/journal, or receive them from the other logging system over the network. Then, the logs may be stored in the local files in the /var/log directory, or sent to Elasticsearch, or forwarded to other logging system. The combination between the inputs and the outputs needs to be flexible. For instance, you may want to inputs from journal stored just in the local file, while inputs read from files stored in the local log files as well as forwarded to the other logging system.
To satisfy such requirements, logging role introduced 3 primary
variables logging_inputs
, logging_outputs
, and logging_flows
. The
inputs are represented in the list of logging_inputs
dictionary, the
outputs are in the list of logging_outputs
dictionary, and the
relationship between them are defined as a list of logging_flows
dictionary. The details are described in
Logging Configuration.
Requirements
This role is supported on RHEL/CentOS-7, RHEL/CentOS-8 and Fedora distributions.
Definitions
-
logging_inputs
- List of logging inputs dictionary to specify input types.-
basics
- basic inputs configuring inputs from systemd journal or unix socket. -
files
- files inputs configuring inputs from local files. -
remote
- remote inputs configuring inputs from the other logging system over network. -
ovirt
- ovirt inputs configuring inputs from the oVirt system.
-
-
logging_outputs
- List of logging outputs dictionary to specify output types.-
elasticsearch
- elasticsearch outputs configuring outputs to elasticsearch. It is available only when the input isovirt
. -
files
- files outputs configuring outputs to the local files. -
forwards
- forwards outputs configuring outputs to the other logging system. -
remote_files
- remote files outputs configuring outputs from the other logging system to the local files.
-
-
logging_flows
- List of logging flows dictionary to define relationships between logging inputs and outputs. -
Rsyslog
- The logging role default log provider used for log processing. -
Elasticsearch
- Elasticsearch is a distributed, search and analytic engine for all types of data. One of the supported outputs in the logging role.
Logging Configuration
Brief overview
Logging role allows to have variables logging_inputs
,
logging_outputs
, and logging_flows
with additional options to
configure logging system such as rsyslog
.
Currently, the logging role supports four types of logging
inputs: basics
, files
, ovirt
, and remote
.
And four types of outputs: elasticsearch
,
files
, forwards
, and remote_files
. To deploy configuration files
with these inputs and outputs, specify the inputs as logging_inputs
and the outputs as logging_outputs
. To define the flows from inputs to
outputs, use logging_flows
. The logging_flows
has three keys name
,
inputs
, and outputs
, where inputs
is a list of
logging_inputs name
values and outputs
is a list of
logging_outputs name
values.
This is a schematic logging configuration to show log messages from input_nameA are passed to output_name0 and output_name1; log messages from input_nameB are passed only to output_name1.
---
- name: a schematic logging configuration
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: input_nameA
type: input_typeA
- name: input_nameB
type: input_typeB
logging_outputs:
- name: output_name0
type: output_type0
- name: output_name1
type: output_type1
logging_flows:
- name: flow_nameX
inputs: [input_nameA]
outputs: [output_name0, output_name1]
- name: flow_nameY
inputs: [input_nameB]
outputs: [output_name1]
Variables
Logging_inputs options
logging_inputs
: A list of following dictionary to configure inputs.
-
common keys
-
name
: Unique name of the input. Used in thelogging_flows
inputs list and a part of the generated config filename. -
type
: Type of the input element. Currently,basics
,files
,ovirt
, andremote
are supported. Thetype
is used to specify a task type which corresponds to a directory name in roles/rsyslog/{tasks,vars}/inputs/. -
state
: State of the configuration file.present
orabsent
. Default topresent
.
-
-
basics
type -basics
input supports reading logs from systemd journal or systemd unix socket. available options-
kernel_message
: Loadimklog
if set totrue
. Default tofalse
. -
use_imuxsock
: Useimuxsock
instead ofimjournal
. Default tofalse
. -
ratelimit_burst
: Maximum number of messages that can be emitted within ratelimit_interval. Default to20000
if use_imuxsock is false. Default to200
if use_imuxsock is true. -
ratelimit_interval
: Interval to evaluate ratelimit_burst. Default to600
seconds if use_imuxsock is false. Default to0
if use_imuxsock is true. 0 indicates ratelimiting is turned off. -
persist_state_interval
: Journal state is persisted every value messages. Default to10
. Effective only when use_imuxsock is false.
-
-
files
type -files
input supports reading logs from the local files. available options-
input_log_path
: File name to be read by the imfile plugin. The value should be full path. Wildcard * is allowed in the path. Default to/var/log/containers/*.log
.
-
-
ovirt
type -ovirt
input supports oVirt specific inputs. For the details, visit oVirt Support. -
remote
type -remote
input supports receiving logs from the remote logging system over the network. This input type makes rsyslog a server. available options-
udp_port
: UDP port number to listen. Default to514
. -
tcp_port
: TCP port number to listen. Default to514
. -
tcp_tls_port
: TLS TCP number to listen. Default to6515
. To use tcp_tls_port,logging_pki
andlogging_pki_files
need to be configured. They are described in Other variables.
-
Logging_outputs options
logging_outputs
: A list of following dictionary to configure outputs.
-
common keys
-
name
: Unique name of the output. Used in thelogging_flows
outputs list and a part of the generated config filename. -
type
: Type of the output element. Currently,elasticsearch
,files
,forwards
, andremote_files
are supported. Thetype
is used to specify a task type which corresponds to a directory name in roles/rsyslog/{tasks,vars}/outputs/. -
state
: State of the configuration file.present
orabsent
. Default topresent
.
-
-
elasticsearch
type -elasticsearch
output supports sending logs to Elasticsearch. It is available only when the input isovirt
. Assuming Elasticsearch is already configured and running. available options-
server_host
: Host name Elasticsearch is running on. Required. -
server_port
: Port number Elasticsearch is listening to. Default to9200
. -
index_prefix
: Elasticsearch index prefix the particular log will be indexed to. Required. -
input_type
: Specifying the input type. Currently only typeovirt
is supported. Default toovirt
. -
retryfailures
: Specifying whether retries or not in case of failure. Allowed value istrue
orfalse
. Default totrue
. -
use_cert
: If true, key/certificates are used to access Elasticsearch. Triplets {ca_cert
,cert
,key
} and/or {ca_cert_src
,cert_src
,key_src
} should be configured. Default totrue
. -
ca_cert
: Path to CA cert for Elasticsearch. Default to/etc/rsyslog.d/es-ca.crt
-
cert
: Path to cert to connect to Elasticsearch. Default to/etc/rsyslog.d/es-cert.pem
. -
key
: Path to key to connect to Elasticsearch. Default to/etc/rsyslog.d/es-key.pem
. -
ca_cert_src
: Local CA cert file path which is copied to the target host. Ifca_cert
is specified, it is copied to the location. Otherwise, to logging_config_dir. -
cert_src
: Local cert file path which is copied to the target host. Ifcert
is specified, it is copied to the location. Otherwise, to logging_config_dir. -
key_src
: Local key file path which is copied to the target host. Ifkey
is specified, it is copied to the location. Otherwise, to logging_config_dir.
-
-
files
type -files
output supports storing logs in the local files usually in /var/log. available options-
facility
: Facility; default to*
. -
severity
: Severity; default to*
. -
exclude
: Exclude list; default to none. -
path
: Path to the output file.Unless the above options are given, these local file outputs are configured.
kern.* /dev/console *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron *.emerg :omusrmsg:* uucp,news.crit /var/log/spooler local7.*
-
-
forwards
type -forwards
output sends logs to the remote logging system over the network. This is for the client rsyslog. available options-
facility
: Facility; default to*
. -
severity
: Severity; default to*
. -
protocol
: Protocol;tcp
orudp
; default totcp
. -
target
: Target host (fqdn). Required. -
port
: Port; default to 514.
-
-
remote_files
type -remote_files
output stores logs to the local files per remote host and program name originated the logs. available options-
facility
: Facility; default to*
. -
severity
: Severity; default to*
. -
exclude
: Exclude list; default to none. -
async_writing
: If set totrue
, the files are written asynchronously. Allowed value istrue
orfalse
. Default tofalse
. -
client_count
: Count of client logging system supported this rsyslog server. Default to10
. -
io_buffer_size
: Buffer size used to write output data. Default to65536
bytes. -
remote_log_path
: Full path to store the filtered logs. This is an example to support the per host output log files/path/to/output/dir/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log
-
remote_sub_path
: Relative path to logging_system_log_dir to store the filtered logs.if both
remote_log_path
andremote_sub_path
are not specified, the remote_file output configured with the following settings.template( name="RemoteMessage" type="string" string="/var/log/remote/msg/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) template( name="RemoteHostAuthLog" type="string" string="/var/log/remote/auth/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) template( name="RemoteHostCronLog" type="string" string="/var/log/remote/cron/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) template( name="RemoteHostMailLog" type="string" string="/var/log/remote/mail/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log" ) ruleset(name="unique_remote_files_output_name") { authpriv.* action(name="remote_authpriv_host_log" type="omfile" DynaFile="RemoteHostAuthLog") *.info;mail.none;authpriv.none;cron.none action(name="remote_message" type="omfile" DynaFile="RemoteMessage") cron.* action(name="remote_cron_log" type="omfile" DynaFile="RemoteHostCronLog") mail.* action(name="remote_mail_service_log" type="omfile" DynaFile="RemoteHostMailLog") }
-
Logging_flows options
-
name
: Unique name of the flow. -
inputs
: A list of inputs, from which processing log messages starts. -
outputs
: A list of outputs. to which the log messages are sent.
Other variables
These variables are set in the same level of the logging_inputs
,
logging_output
, and logging_flows
.
-
logging_enabled
: When true, logging role will deploy specified configuration file set. Default to true. -
logging_pki
: Specifying an encryption. One ofnone
,ptcp
,tls
,gtls
, andgnutls
. Default toptcp
. Note:none
=ptcp
,tls
=gtls
=gnutls
. Whenlogging_pki
is notptcp
,rsyslog_pki_path
,rsyslog_pki_realm
,rsyslog_pki_ca
,rsyslog_pki_crt
,rsyslog_pki_key
are configured. -
logging_pki_files
: Specifying a list of ca_cert, cert and key dictionary to specify the source of the files if any as well as the destination, which is set in the deployed config file. The configuration is used by theremote
input andforward
output.
- type: ca_cert | cert | key
src: location of the file on the local host;
if given, the file is deployed to dest value path.
dest: path to be deployed on the target host;
if given, the path is set to the config file.
Default to /etc/pki/tls/certs/ca.crt for ca_cert
/etc/pki/tls/certs/cert.pem for cert
/etc/pki/tls/private/key.pem for key
Security parameters
-
logging_pki_authmode
: Specify the default network driver authentication mode.x509/name
oranon
are available: Default tox509/name
. -
logging_domain
: The default DNS domain used to accept remote incoming logs from remote hosts. Default to "{{ ansible_domain if ansible_domain else ansible_hostname }}" -
logging_permitted_peers
: List of hostnames, IP addresses or wildcard DNS domains which will be allowed by thelogging
server to connect and send logs over TLS. Default to [*.{{ logging_domain }}] -
logging_send_permitted_peers
: List of hostnames, IP addresses or wildcard DNS domains which will be verified by thelogging
client and will allow to send logs to the remote server over TLS. Default to "{{ logging_permitted_peers }}".
Server Performance optimization
-
logging_tcp_threads
: Input thread count listening on the tcp port. Default to1
. -
logging_udp_threads
: Input thread count listening on the udp port. Default to1
. -
logging_udp_system_time_requery
: Everyvalue
OS system calls, get the system time. Recommend not to set above 10. Default to2
times. -
logging_udp_batch_size
: Maximum number of udp messages per OS system call. Recommend not to set above 128. Default to32
. -
logging_server_queue_type
: Type of queue.FixedArray
is available. Default toLinkedList
. -
logging_server_queue_size
: Maximum number of messages in the queue. Default to50000
. -
logging_server_threads
: Number of worker threads. Default tologging_tcp_threads
+logging_udp_threads
. -
logging_mark
: Mark message periodically by immark, if set totrue
. Default tofalse
. -
logging_mark_interval
: Interval forlogging_mark
in seconds. Default to3600
. -
logging_purge_original_conf
:true
orfalse
. If set totrue
, files in /etc/rsyslog.d are purged. -
logging_system_log_dir
: Directory where the local log output files are placed. Default to/var/log
.
Update and Delete
Due to the nature of ansible idempotency, if you run ansible-playbook multiple times without changing any variables and options, no changes are made from the second time. If some changes are made, only the rsyslog configuration files affected by the changes are recreated. To delete any existing rsyslog input or output config files generated by the previous ansible-playbook run, you need to add "state: absent" to the dictionary to be deleted (in this case, input_nameA and output_name0). And remove the flow dictionary related to the input and output as follows.
logging_inputs:
- name: input_nameA
type: input_typeA
state: absent
- name: input_nameB
type: input_typeB
logging_outputs:
- name: output_name0
type: output_type0
state: absent
- name: output_name1
type: output_type1
logging_flows:
- name: flow_nameY
inputs: [input_nameB]
outputs: [output_name1]
Configuration Examples
Standalone configuration
-
Deploying
basics input
reading logs from systemd journal and implicitfiles output
to write to the local files.
---
- name: Deploying basics input and implicit files output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: system_input
type: basics
The following playbook generates the same logging configuration files.
---
- name: Deploying basics input and files output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: system_input
type: basics
logging_outputs:
- name: files_output
type: files
logging_flows:
- name: flow0
inputs: [system_input]
outputs: [files_output]
-
Deploying
basics input
reading logs from systemd unix socket andfiles output
to write to the local files.
---
- name: Deploying basics input using systemd unix socket and files output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: system_input
type: basics
use_imuxsock: true
logging_outputs:
- name: files_output
type: files
logging_flows:
- name: flow0
inputs: [system_input]
outputs: [files_output]
-
Deploying
basics input
reading logs from systemd journal andfiles output
to write to the individually configured local files.
---
- name: Deploying basic input and configured files output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: system_input
type: basics
logging_outputs:
- name: files_output0
type: files
severity: info
exclude:
- authpriv.none
- auth.none
- cron.none
- mail.none
path: /var/log/messages
- name: files_output1
type: files
facility: authpriv,auth
path: /var/log/secure
logging_flows:
- name: flow0
inputs: [system_input]
outputs: [files_output0, files_output1]
-
Deploying
files input
reading logs from local files andfiles output
to write to the individually configured local files.
---
- name: Deploying files input and configured files output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: files_input0
type: files
input_log_path: /var/log/containerA/*.log
- name: files_input1
type: files
input_log_path: /var/log/containerB/*.log
logging_outputs:
- name: files_output0
type: files
severity: info
exclude:
- authpriv.none
- auth.none
- cron.none
- mail.none
path: /var/log/messages
- name: files_output1
type: files
facility: authpriv,auth
path: /var/log/secure
logging_flows:
- name: flow0
inputs: [files_input0, files_input1]
outputs: [files_output0, files_output1]
-
Deploying
files input
reading logs from a local file andelasticsearch output
to store the logs. Assuming the ca_cert, cert and key to connect to Elasticsearch are prepared.
---
- name: Deploying basic input and elasticsearch output
hosts: all
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: files_input
type: files
input_log_path: /var/log/containers/*.log
logging_outputs:
- name: elasticsearch_output
type: elasticsearch
server_host: your_target_host
server_port: 9200
index_prefix: project.
input_type: ovirt
ca_cert_src: /local/path/to/ca_cert
cert_src: /local/path/to/cert
key_src: /local/path/to/key
logging_flows:
- name: flow0
inputs: [files_input]
outputs: [elasticsearch_output]
Client configuration
-
Deploying
basics input
reading logs from systemd journal andforwards output
to forward the logs to the remote rsyslog.
---
- name: Deploying basics input and forwards output
hosts: clients
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: basic_input
type: basics
logging_outputs:
- name: forward_output0
type: forwards
severity: info
protocol: udp
target: your_target_hostname
port: 514
- name: forward_output1
type: forwards
facility: mail
protocol: tcp
target: your_target_hostname
port: 514
logging_flows:
- name: flows0
inputs: [basic_input]
outputs: [forward_output0, forward_output1]
-
Deploying
files input
reading logs from a local file andforwards output
to forward the logs to the remote rsyslog over tls. Assuming the ca_cert, cert and key files are prepared. The files are deployed to the default location/etc/pki/tls/certs/
,/etc/pki/tls/certs/
, and/etc/pki/tls/private
, respectively.
---
- name: Deploying files input and forwards output with certs
hosts: clients
roles:
- linux-system-roles.logging
vars:
logging_pki: tls
logging_pki_files:
- type: ca_cert
src: /local/path/to/ca_cert
- type: cert
src: /local/path/to/cert
- type: key
src: /local/path/to/key
logging_inputs:
- name: files_input
type: files
input_log_path: /var/log/containers/*.log
logging_outputs:
- name: forwards_output
type: forwards
protocol: tcp
target: your_target_host
port: 6514
logging_flows:
- name: flows0
inputs: [basic_input]
outputs: [forwards-severity_and_facility]
Server configuration
-
Deploying
remote input
reading logs from remote rsyslog andremote_files output
to write the logs to the local files under the directory named by the remote host name.
---
- name: Deploying remote input and remote_files output
hosts: server
roles:
- linux-system-roles.logging
vars:
logging_inputs:
- name: remote_udp_input
type: remote
udp_port: 514
- name: remote_tcp_input
type: remote
tcp_port: 514
logging_outputs:
- name: remote_files_output
type: remote_files
logging_flows:
- name: flow_0
inputs: [remote_udp_input, remote_tcp_input]
outputs: [remote_files_output]
-
Deploying
remote input
reading logs from remote rsyslog andremote_files output
to write the logs to the configured local files with the tls setup supporting 20 clients. Assuming the ca_cert, cert and key files are prepared. The files are deployed to the default location/etc/pki/tls/certs/
,/etc/pki/tls/certs/
, and/etc/pki/tls/private
, respectively.
---
- name: Deploying remote input and remote_files output with certs
hosts: server
roles:
- linux-system-roles.logging
vars:
logging_pki: tls
logging_pki_files:
- type: ca_cert
src: /local/path/to/ca_cert
- type: cert
src: /local/path/to/cert
- type: key
src: /local/path/to/key
logging_inputs:
- name: remote_tcp_input
type: remote
tcp_tls_port: 6514
logging_outputs:
- name: remote_files_output0
type: remote_files
remote_log_path: /var/log/remote/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log
async_writing: true
client_count: 20
io_buffer_size: 8192
- name: remote_files_output1
type: remote_files
remote_sub_path: others/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log
logging_flows:
- name: flow_0
inputs: [remote_udp_input, remote_tcp_input]
outputs: [remote_files_output0, remote_files_output1]
Providers
Rsyslog - This documentation contains rsyslog specific information.
Tests
Automated CI-tests - This documentation shows how to execute CI tests in the tests directory as well as how to debug when the test fails.
Manual tests - This documentation shows how to run the logging role manually against the local host or a remote host.