The /var/log/watchmaker/salt_call.debug.log Log-File

This is the log-file that captures the bulk of the SaltStack-related state-output. This file gets created when watchmaker has been able to successfully download all of its execution information. This file gets created shortly after this line appears in the /var/log/watchmaker/watchmaker.log file:

2023-06-15 11:13:27,378 [watchmaker.workers.base.SaltLinux][DEBUG][6407]: Command: /usr/bin/salt-call --local --retcode-passthrough --no-color --config-dir /opt/watchmaker/salt --log-file /var/log/watchmaker/salt_call.debug.log --log-file-level debug --log-level error --out quiet --return local state.highstate

Typically, the only errors that will appear here are the results of errors in the SaltStack formulae for the standard integrations. To see which modules may get logged into this file, look at the contents of the /srv/watchmaker/salt/formulas/ directory and then cross-reference those directories against the contents of the /srv/watchmaker/salt/states/top.sls file. To help interpret, a typical top.sls file’s contents is offered:

{%- set environments = ['dev', 'test', 'prod', 'dx'] %}

base:
  'G@os_family:RedHat':
    - name-computer
    - scap.content
    - ash-linux.vendor
    - ash-linux.stig
    - ash-linux.iavm
{%- if salt.grains.get('watchmaker:enterprise_environment') | lower in environments %}
    - join-domain
    - mcafee-agent
    - splunkforwarder
    - nessus-agent.elx.install
    # Recommend other custom states be inserted here
{%- endif %}
    - scap.scan

  'G@os_family:Windows':
    [...elided...]

In the above, these salt formulas will be executed unconditionally on RedHat-derivative systems:

• /srv/watchmaker/salt/formulas/name-computer-formula

• /srv/watchmaker/salt/formulas/ash-linux-formula[1]

Similarly, the contents of the following directories will be executed by watchmaker only if the environment specified in the watchmaker-invocation (the string-value after the -e flag) matches one of the elements in the environments list.

• /srv/watchmaker/salt/formulas/join-domain-formula

• /srv/watchmaker/salt/formulas/mcafee-agent-formula

• /srv/watchmaker/salt/formulas/nessus-agent-formula

• /srv/watchmaker/salt/formulas/splunkforwarder-formula

Similarly, the behavior of each of the above states’ executions will be governed by content specified under the /srv/watchmaker/salt/pillar directory hierarchy. This content is used to feed values into the parameter-driven SaltStack states enumerated in the .../formulas directories.

Typical Error Causes

The most frequent causes of errors, once watchmaker has caused Saltstack states to begin their execution, are errors encountered while running the individual enterprise-integration states. Typically, these errors are around stale configuration data (expired domain-join credentials for directory-integration or stale host/IP/port information for other services) or communication-issues between the OS that watchmaker is configuring and the service watchmaker is attempting to configure the instance to integrate: DNS resolution, host or network-level firewall rules, other transit-issues.

The next most frequent errors are already-existing configuration problems in the OS that watchmaker is configuring. These include things like:

• Failures accessing RPM repositories (especially problematic with repositories that require client-cert authentication where there are certificate-expiration problems between the RPM client and repository server)

• Too little storage in critical partitions

• The watchmaker activities running after something else has changed a resource-configuration that watchmaker expects to manage but finds the resource in an unanticipated state

The least frequent cause of errors is related to the SaltStack code itself. Usually, this is caught in pre-release testing, but “bugs happen”. While states are typically coded to try to gracefully handle errors encountered – they’ll typically still fail, but at least try to provide meaningful error-output. Usually, the “bugs happen” errors are resultant of environment-to-environment deltas that were not adequately specified to the code-maintainers or the requisite logic-branching was not able to be adequately exercised across the various environments.

For errors in enterprise-integration content, efforts have been undertaken to try to ensure those errors are adequately represented in this log-file. However, the application-specific logs (the ones for the integrated-application) will still remain the authoritative source for troubleshooting exercises.

---

[1]

Due to the ash-linux.vendor, ash-linux.stig and ash-linux.iavm specification, only the ash-linux-formula’s vendor, stig and iavm states’ executions will be attempted.