--- layout: default title: Reporting published: false sorting: 80 tags: [overviews, special topics, guide] --- # What are monitoring and reporting? Monitoring is the sampling of system variables at regular intervals in order to present an overview of actual changes taking place over time. Monitoring data are often presented as extensive views of moving-line time series. Monitoring has the ability to detect anomalous behaviour by comparing past and present. The term reporting is usually taken to mean the creation of short summaries of specific system properties suitable for management. System reports describe both promises about the system, such as compliance, discovered changes and faults. The challenge of both these activities is to compare intended or promised, behaviour with the actual observed behaviour of the system. # Should monitoring and configuration be separate? The traditional view of IT operations is that configuration, monitoring and reporting are three different things that should not be joined. Traditionally, all three have been independent centralized processes. This view has emerged historically, but it has a major problem. Humans are needed to glue these parts back together. Monitoring as an independent activity is inherently non-scalable. When numbers of hosts grow beyond a few thousands, centralized monitoring schemes fail to manage the information. Tying configuration (and therefore repair) to monitoring at the host level is essential for the effective management of large and distributed data facilities. CFEngine foresaw this need in 1998, with its Computer Immunology initiative, and continues to develop this strategy. CFEngine's approach is to focus on scalability. The commercial editions of CFEngine provide what meaningful information they can in a manner that can be scaled to tens of thousands of machines. # Reporting in CFEngine If you have regular reporting needs, we recommend using our commercially supported version of CFEngine (CFEngine Nova or above), as you will save considerable time and resources in programming, and you will have access to the latest developments through the software subscription. No promises made in CFEngine imply automatic aggregation of data to a central location. In commercial CFEngine versions, e.g. CFEngine Nova, an optimized aggregation of standardized reports is provided, but the ultimate decision to aggregate must be yours. Monitoring and reporting capabilities in CFEngine depend on the software version include: * Community Edition: Basic output to file or logs may be customized on a per-promise basis. Users can design their own log and report formats, but data processing and extraction from CFEngine's embedded databases must be scripted by the user. * Nova: In addition to community features, Nova/Enterprise provides automated extraction of data from CFEngine's self-learning agents, and the generation of a standard set of reports in text, HTML or XML formats. Nova summarizes distributed data and provides simple compression and aggregation of these summaries. Finally summaries are tied into a knowledge map or semantic index for browsing by IT operations. Command line tools in cf-report are also available for Nova users to browse network-wide data. # Standard reports in CFEngine The following list of reports are only available in full in commercial editions of CFEngine. Some sample reports are provided in the Community Edition. * Available patches report Patches already installed on system if available. * Classes report User defined classes observed on the system – inventory data. * Compliance report Total summary of host compliance, all promises aggregated over time. * File_changes report Latest observed changes to system files with time discovered. * File_diffs report Latest observed differences to system files, in a simple diff format. * Hashes report File hash values measured (change detection). * Installed patches report Patches not yet installed, but published by vendor if available. * Installed software report Software already installed on system if available. * Lastseen report Time and frequency of communications with peers, host reliability. * Micro-audit report Generated by CFEngine self-auditing. This report is not aggregated. * Monitor summary report Pseudo-real-time measurement of time series data. * Performance report Time cost of verifying system promises. * Promise report Per-promise average compliance report over time. * Promises not kept report Promises that were recently un-kept. * Promises repaired report Promises that were recently kept by repairing system state. * Setuid report Known setuid programs found on system. * Variables report Current variable values expanded on different hosts. # CFEngine output levels CFEngine's default behaviour is to report to the console (known as standard output). It's default behaviour is to report nothing except errors that are judged to be of a critical nature. By using CFEngine with the inform flag: ```console # cf-agent -I # cf-agent --inform ``` You can alter the default to report on action items (actual changes) and warnings by using CFEngine with the verbose flag: ```console # cf-agent -v # cf-agent --verbose ``` You can alter the default to report all of its thought-processes. You should not interpret a message that only appears in CFEngine's verbose mode as an actual error, only as information that might be relevant to decisions being made by the agent. # Creating custom reports – all versions CFEngine allows you to usereportspromises to make reports of your own. A simple example of this is shown below. ```cf3 body common control { bundlesequence => { "test" }; } # bundle agent test { reports: cfengine_3:: "$(sys.date),This is a report" report_to_file => "/tmp/test_log"; } ``` We can apply this idea to make more useful custom reports. In this example, the agent tests for certain software package and creates a simple HTML file of existing software. ```cf3 body common control { bundlesequence => { "test" }; } # bundle agent test { vars: "software" slist => { "gpg", "zip", "rsync" }; classes: "no_report" expression => fileexists("/tmp/report.html"); "have_$(software)" expression => fileexists("/usr/bin/$(software)"); reports: no_report:: " Name of this host is: $(sys.host)
Type of this host is: $(sys.os)
" report_to_file => "/tmp/report.html"; # " Host has software $(software)
" if => "have_$(software)", report_to_file => "/tmp/report.html"; # " " report_to_file => "/tmp/report.html"; } ``` The outcome of this promise is a file called /tmp/report.html containing output like this: ``` Name of this host is: atlas
Type of this host is: linux
Host has software gpg
Host has software zip
Host has software rsync
``` The mechanism shown above, can clearly be used to create a wide variety of report formats, but it requires a lot of coding and maintenance by the user. CFEngine Nova simplifies this kind of report generation by enabling and updating many out-of-the-box reports directly from the cf-report agent. # Including data in reports CFEngine generates information internally that you might want to use in reports. For example, the agent cf-agent interfaces with the local light-weight monitoring agent cf-monitord so that system state can be reported simply: ```cf3 body common control { bundlesequence => { "report" }; } ########################################################### bundle agent report { reports: linux:: "/etc/passwd except $(const.n)" showstate => { "otherprocs", "rootprocs" }; } ``` A corollary to this is that you can get CFEngine to report system anomalies. ``` reports: rootprocs_high_dev2:: "RootProc anomaly high 2 dev on $(mon.host) at approx $(mon.env_time) measured value $(mon.value_rootprocs) average $(mon.average_rootprocs) pm $(mon.stddev_rootprocs)" showstate => { "rootprocs" }; entropy_www_in_high&anomaly_hosts.www_in_high_anomaly:: "High entropy incoming www anomaly on $(mon.host) at $(mon.env_time) measured value $(mon.value_www_in) average $(mon.average_www_in) pm $(mon.stddev_www_in)" showstate => { "incoming.www" }; ``` This produces standard output of the form: ``` R: State of otherprocs peaked at Tue Dec 1 12:12:21 2009 R: The peak measured state was q = 98: R: Frequency: [kjournald] |** (2/98) R: Frequency: [pdflush] |** (2/98) R: Frequency: /var/cfengine/bin/cf-execd|** (2/98) R: Frequency: COMMAND |* (1/98) R: Frequency: init [5] |* (1/98) R: Frequency: [kthreadd] |* (1/98) R: Frequency: [migration/0] |* (1/98) R: Frequency: [ksoftirqd/0] |* (1/98) R: Frequency: [events/0] |* (1/98) R: Frequency: [khelper] |* (1/98) R: Frequency: [kintegrityd/0] |* (1/98) ``` Finally, you can quote lines from files in your data for convenience. ```cf3 body common control { bundlesequence => { "report" }; } ########################################################### bundle agent report { reports: linux:: "/etc/passwd except $(const.n)" printfile => pr("/etc/passwd","5"); } ###################################################################### body printfile pr(file,lines) { file_to_print => "$(file)"; number_of_lines => "$(lines)"; } ``` This produces output of the form ``` R: /etc/passwd except R: at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash R: avahi:x:103:105:User for Avahi:/var/run/avahi-daemon:/bin/false R: beagleindex:x:104:106:User for Beagle indexing:/var/cache/beagle:/bin/bash R: bin:x:1:1:bin:/bin:/bin/bash R: daemon:x:2:2:Daemon:/sbin:/bin/bash ``` # Creating custom logs Logs can be attached to any promise. In this example, an executed shell command logs a message to the standard output. CFEngine recognizes thestdoutfilename for Standard Output, in the Unix/C standard manner. ```cf3 bundle agent test { commands: "/tmp/myjob", action => logme("executor"); } ############################################ body action logme(x) { log_repaired => "stdout"; logstring => " -> Started the $(x) (success)"; } ``` In this next example, a file creation promise logs different outcomes (success or failure) to different log files. ```cf3 body common control { bundlesequence => { "test" }; } bundle agent test { vars: "software" slist => { "/root/xyz", "/tmp/xyz" }; files: "$(software)" create => "true", action => logme("$(software)"); } # body action logme(x) { log_kept => "/tmp/private_keptlog.log"; log_failed => "/tmp/private_faillog.log"; log_repaired => "/tmp/private_replog.log"; log_string => "$(sys.date) $(x) promise status"; } ``` This generates three different logs with outputs in of the form: ```console atlas$ more /tmp/private_keptlog.log Sun Dec 6 11:58:16 2009 /tmp/xyz promise status Sun Dec 6 11:58:43 2009 /tmp/xyz promise status ``` # Redirecting output to logs CFEngine interfaces with the system logging tools in different ways. Syslog is the default log for Unix-like systems, while the event logger is the default on Windows. You may choose to copy a fixed level of CFEngine's standard screen messaging to the system logger on a per-promise basis. ```cf3 body common control { bundlesequence => { "one" }; } bundle agent one { files: "/tmp/xyz" create => "true", action => log; } body action log { log_level => "inform"; } ``` # Change auditing - the all seeing eye Total auditing of a system is a surprisingly difficult thing to do, and it is extremely resource intensive. The followers of an audit trail are often paranoid by nature and are seldom satisfied with the level of detail they find. However, the times we really need an audit are rare, but the cost is ever present. The price of certainty is high. Spend a moment considering this: if you want to describe every change of state that happens on a computer, then you need to remember old state and compare it to new state. Then you have to record the differences. So you need more than the entire size of your computer's normal resources to do this. Your storage efficiency will always be less than 50% and your processing efficiency will be less than 50% on every audited item. Is this worth the effort? Perhaps your resources would be better spent keeping targeted backups and simply rebuilding contaminated systems. Switch on auditing like this: ```cf3 body agent control { auditing => "true"; } ``` If you decide to go for full auditing, CFEngine will not collect and centralize the reports as they will be too large for this to be a scalable operation. Still, you can view them in a web browser on the local host, or copy them manually to a suitable location. # Cheaper options - tripwires Doing a change detection scan is a convergent process, but it can still detect changes and present the data in a compressed format that is often more convenient than auditing. The result is less precise, but there is a trade-off between precision and cost. To make a change tripwire, you use a files promise, something like this: ```cf3 body common control { bundlesequence => { "testbundle" }; } # bundle agent testbundle { files: "/home/mark/tmp" -> "me" changes => scan_files, depth_search => recurse("inf"); } # library code ... body changes scan_files { report_changes => "all"; update_hashes => "true"; } body depth_search recurse(d) { depth => "$(d)"; } ``` In CFEngine Nova, reports of the following form are generated when these promises are kept by the agent: ``` Change detected File change Sat Dec 5 18:27:44 2009 group for /tmp/testfile changed 100 -> 0 Sat Dec 5 18:27:44 2009 /tmp/testfile Sat Dec 5 18:20:45 2009 /tmp/testfile ``` These reports are generated automatically in CFEngine Nova, and are integrated into the web browsable knowledge map. Community edition users have to extract the data and create these themselves. # Commercial edition measurements promises In commercial versions of CFEngine, you can extract data from the system in more sophisticated ways from files or pipes, using Perl Compatible Regular Expressions to match text. Thecf-monitordagent is responsible for processing measurement promises. In this example, we count lines matching a pattern in a file. You might want to scan a log for instances of a particular message and trace this number over time. ```cf3 bundle monitor watch { measurements: "/tmp/file" handle => "line_counter", stream_type => "file", data_type => "counter", match_value => scanlines("MYLINE.*"), history_type => "log"; } # body match_value scanlines(x) { select_line_matching => "^$(x)$"; } ``` See the CFEngine Nova documentation for more possibilities of measurement promises. # Hub Reporting In the commercial editions of CFEngine much more extensive and searchable reporting is available. # Mission Portal access to the hub The preferred approach to querying information on a hub is to use the web interface in the Mission Portal. This gives the greatest flexibility in both search and presentation of data. Given the extensiveness of the Mission Portal user interface, the details are covered in a separate document. # Command line access to the hub Users with login access to the hub can also use the command line tool cf-report to extract a limited view of the data. Currently supported reports include: * compliance The percentage total compliance log for all hosts. * dead-clients Shows a list of client hosts that have not made incoming requests within the standard time horizon (default 15 minutes). * file_changes The change log * file_diffs The change details for text files. * last-seen Show the last time hosts connnected to the hub * promises Compliance by promise, labelled by promise-handle. * setuid The list of setuid/setgid root files detected on the system. * software The installed software base of the system. * summary A summary of how many hosts are compliant within a given set of search parameters. * vars The values of variables set on hosts. Some special command line options are supported in the commercial versions. * ‘--query-hub’ or ‘-q’ Query hub database interactively. This option is the entry point for querying the hub data with cf-report, and must always be specified. * ‘--show name’ Select the name of the report from the above list. * ‘--promise-handle’ or ‘-p regex’. For promise compliance report, this defines a regular expression to search for promises of a specific name. Specify a promise-handle to look up * ‘--hostkey’ or ‘-k hashkey’. Specify a particular host to query for data, using the unique host-key. * ‘--class-regex’ or ‘-c regex’ - Specify a class regular expression to search for * ‘--filter’ or ‘-F regex’ - Specify a name regular expression for filtering results # Example command hub searches If only a host-key is specified, CFEngine returns with the last known location and identity of the host. (Note that, in the following examples, the SHA keys are reduced for readability). ```console host# cf-report -q --hostkey SHA=bd6dfcc2... -> Hostname: hub.test.cfengine.com -> Recent IP Addresses: 10.0.0.29 ``` To dump all values from all hosts: * cf-report -q --show promises * cf-report --query-hub --show promises You can select a single host for a particular report: * cf-report -q --hostkey SHA=c40fb732c6e5... --show vars Or you can select a CFEngine class of hosts that will be selected to report * cf-report -q --show summary --class-regex linux * cf-report -q --show summary --class-regex SuSE * cf-report -q --show summary --class-regex NewYork Here are some examples using filters to 'grep' out certain items: * cf-report -q --hostkey SHA=c40fb732c6... --show vars --filter date * cf-report -q --filter "mail.*" --hostkey SHA=bd6dfccb1a... --show setuid * cf-report -q --show promises -p knowledge_files_db_stamp