```{eval-rst}
.. image:: images/cropped-plus3it-logo-cmyk.png
   :width: 140px
   :alt: Powered by Plus3 IT Systems
   :align: right
   :target: https://www.plus3it.com
```
<br>

# Frequently Asked Questions

## How do I know if watchmaker has installed?

To determine whether watchmaker is installed, the simplest method is to run the
command `watchmaker --help`. If it displays the cli help page, watchmaker is
installed. Another option is to check `pip list | grep watchmaker`.

## What do I do if watchmaker failed to install?

First, review the [installation](installation) document. Then double-check
the output of a failed installation. Usually, the output points pretty clearly
at the source of the problem. Watchmaker can be re-installed over itself with
no problem, so once the root cause is resolved, simply re-install watchmaker.

## Why does watchmaker immediately fail to run on my spel-based system?

System-images created using [spel](https://github.com/plus3it/spel/) enable
SELinux role-transitions for all local/baked-in user accounts. This includes
the provisioning-users created at launchtime by the `cloud-init` utility. These
users are created with the default SELinux context of
`staff_u:staff_r:staff_t:s0-s0:c0.c1023`. Their default role-transition can
result in the user not having enough privileges to run watchmaker. To work
around any watchmaker-killing "permission denied" errors this may cause,
escalate privileges using `sudo -i -r unconfined_r -t unconfined_t`.

Note: this should only impact processes that are not initiated via `systemd`
(i.e., should not impact provisioning-processes kicked off via userData,
CloudFormation or the like).

## Why does the watchmaker install fail if my system is FIPS enabled?

This is primarily a question for Red Hat (and derived distributions). As of this
writing, the `pip` utility in all Red Hat releases up through 7.4.1708, default
to looking for pypi packages signed with MD5 signatures. If you've enabled FIPS
(or are using a build that has FIPS pre-enabled), MD5 is disabled in the kernel
(due to being a weak hashing-method). You can either disable FIPS (not
recommended) or explicitly force `pip` to use a different signature-index. The
latter is detailed in the Linux section of the [usage](usage) document.

## How do I know if watchmaker has completed without errors?

By default, watchmaker will reboot the system after a sucessful execution.
Therefore, if the system reboots, watchmaker executed successfully. If you are
investigating sometime after watchmaker completed, check the logs for errors.
If anything fails, watchmaker will suppress the reboot. (Though note that the
`--no-reboot` flag can be used to suppress the reboot even after a successful
execution.)

You can also test the watchmaker exit code programmatically. If watchmaker
fails, it will return a non-zero exit code. If watchmaker completes
successfully, it will return an exit code of zero. You would typically pass the
`--no-reboot` flag if you intend to test the exit code and determine what to do
from there.

## What do I do if watchmaker failed to complete or completes with errors?

Start by checking the logs generated by watchmaker. The logs are stored in the
directory specified by the `--log-dir` argument. Search the log for entries
that have `[ERROR]`, this will give you a starting point to begin
troubleshooting. Also, if a salt state failed, look for the pattern
`Result: False`. If it is not an obvious or simple issue, feel free to create
an issue on the watchmaker github page. If there is a salt_call.debug.log in
the watchmaker log directory, you can look for `[ERROR]` messages in there as
well. However, this log file can be very noisy and a message with the error
label may not be related to the error you are encountering.

## Does watchmaker support Enterprise Linux 7?

Watchmaker is supported on RedHat 7 and CentOS 7. See the [index](index)
page for a list of all supported operating systems.

## Does watchmaker support Enterprise Linux 8?

Watchmaker is supported on RedHat 8, CentOS 8 Stream, and Oracle Linux 8. See the
[index](index) page for a list of all supported operating systems.

## Does watchmaker support Enterprise Linux 9?

As of today's date (2024-04-10), watchmaker's hardening-modules do not yet
support Enterprise Linux 9 or related distros. This is currently a
to-be-started project-task. Action on support for Enterprise Linux 9-based
distros can be tracked through [ash-linux-formula issue #496](https://github.com/plus3it/ash-linux-formula/issues/496).

## How can I exclude salt states when executing watchmaker?

The Salt worker in Watchmaker supports an `exclude_states` argument. When
present, the value is passed directly to the `exclude` option of the
[salt highstate execution module](https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.state.html#salt.modules.state.highstate).
To use this option with watchmaker from the command line, pass the argument
`--exclude-states <sls_glob>`. For example:

```shell
# Exclude the state "foo" with an exact match
watchmaker --exclude-states foo

# Exclude all state names that begin with "foo"
watchmaker --exclude-states foo*

# Exclude multiple states "foo" and "bar" with an exact match
watchmaker --exclude-states foo,bar
```

## Can I use the underlying salt functionality directly?

Yes, by passing watchmaker's salt configuration directory to the salt command,
using the `-c|--config-dir` argument:

*   Linux: `/opt/watchmaker/salt`
*   Windows: `C:\Watchmaker\salt\conf`

For example:

```shell
# -c|--config-dir
salt-call -c /opt/watchmaker/salt state.show_top
```

## Can I use watchmaker to toggle my RedHat/Centos host's FIPS mode?

For Enterprise Linux, "yes, indirectly". Because watchmaker implements most of
its functionality via [SaltStack](https://saltproject.io/) modules, you can
directly-use the underlying SaltStack functionality to effect the desired
change. This is done from the commandline - as root - by executing:

*   Disable FIPS-mode: `salt-call -c /opt/watchmaker/salt ash.fips_disable`
*   Enable FIPS-mode: `salt-call -c /opt/watchmaker/salt ash.fips_enable`

And then rebooting the system.

## How do I get Watchmaker release/project notifications?

Users may use an RSS reader of their choice to subscribe to the Watchmaker
Release feed to get notifications on Watchmaker releases. The Watchmaker RSS
release feed is https://github.com/plus3it/watchmaker/releases.atom.

Users can also "watch" the GitHub project to receive notifications on all
project activity, https://github.com/plus3it/watchmaker/subscription.

## Why do I get warnings in my system logs about the rsyslog service not being able to connect to the `logcollector` host

Watchmaker leverages the `oscap` utility and associated OSCAP content for a
significant percentage of its hardening-actions. Recent updates to this content
have included (inappropriately) adding:

~~~
# Per CCE-80863-4: Set *.* @@logcollector in /etc/rsyslog.conf
*.* @@logcollector
~~~

To the `/etc/rsyslog.conf` file. The `*.* @@logcollector` line results in the
`rsyslog` service attempting to do remote-logging to the external host
`logcollector`. If no such hostname exists in the hardened-system's DNS domain,
"host not found" error-types will be logged. If using a log-offloader tool
(such as the Splunk forwarder-agent), it is safe to either comment out the `*.*
@@logcollector`. If data _should_ be sent directly from `rsyslog` to a remote
syslog-host, change the string `logcollector` to the FQDN of your site's actual
log-collector host.

## Why isn't the domain-join functionality doing DDNS updates

Starting in early 2023, the default method for joining Linux hosts to Active
Directory domains was changed to leveraging the native, `sssd`-based
integrations. While `sssd` has the capability of doing DDNS updates (and doing
regular refreshes to prevent record-loss due to record-scavenging), it has a
couple of requirements in order to be able to do so. First and foremost is that
the hosts providing DNS service to the Linux system have DDNS enabled for the A
and PTR zones the Linux system is a member of. Similarly, `sssd` defaults to
attempting to use GSS with TSIG for its DDNS update-requests: even if the Linux
system's A and PTR zones are DDNS-enabled, if those zones don't allow DDNS
clients to negotiate updates with GSS and TSIG, the updates will fail.
Watchmaker's `join-domain-formula` (for `sssd`) _does_ allow overriding the
attempt to negotiate updates with GSS and TSIG. This is done by appending
`dyndns_auth: none` (and, optionally, adding `dyndns_auth_ptr: none`) to the
Salt pillar's `sssd_conf_parameters` parameter-value.

## Why are my security-scans failing due to lack of `nosuid`/`noguid` mount-options for `/boot`

Depending on how your OS was provisioned, the `/boot` directory-tree may or may not be on its own partition. If it's not on its own partition, it will be part of the `/` partition. As a result, it will not be possible to set partition-specific mount-options.

Note: it is known that spel AMIs created for EL8 prior to April of 2024 will
typically _not_ have `/boot` on its own partition. Suport for EFI-boot was only
added to the EL8 automation in February of 2024 and only implemented (in some
regions) in April of 2024. spel AMIs not built to support EFI-boot typically
will not have `/boot` on its own partition.

## How can I see what hardening-actions are implemented through `oscap`

Watchmaker's use of `oscap` (for Linux hosts) is not transparently-implemented.
While the `oscap` activities _are_ logged to the `/var/log/oscap.log` file,
that file's logged-activity is created at an informationl level rather than a
detailed, "debug"-style level. In order to see _exactly_ what the `oscap`
utility does, one can make `oscap` generate a script-file containing all of its
hardening-actions. This can be done by doing (as the `root` user):

~~~bash
oscap xccdf generate fix \
  --fix-type bash \
  --output ~/fixes.sh \
  --profile xccdf_org.ssgproject.content_profile_stig
  /root/scap/content/openscap/ssg-<PLATFORM>-ds.xml
~~~

Where "PLATFORM" will be something like `rhel7` on a Red Hat l system or `ol8`
on an Oracle Linux 8 system. Once the `~/fixes.sh` script is generated, one can
`grep` it for specific actions or open it up for viewing and peruse it for
(presumably problematic) hardening actions (such as the `logcollector` setting
noted in a prior section of this FAQ).