Skip to content

Documentation

Fix hosts automatically

This guide explains remediation: commands that LEA Platform sends to a host to fix a failing rule. It shows you how to turn remediation on safely, how to write a safe fix, where to see the outcome and how to turn it off again.

Before you start

  • You need an administrator account.
  • Read Write and edit rules first; remediation is part of a rule.
  • Have a small test group with a few hosts you can afford to change. See Groups.

Warning: Remediation changes your hosts without asking. A wrong command runs on every host where the rule fails, again and again. Always test on a test group first.

How remediation works

A rule's remediation (also called remedy) is a list of commands, like the validation commands, with Systems and Target Distributions.

  1. The LEA agent runs the rule's validation and sends the output.
  2. The server decides FAIL.
  3. Right away, the server sends the remedy commands that match the host's system to the agent. The agent runs them, and right after them the rule's validation commands again.
  4. The server checks the output of both (remedy output first) with the rule's own expectations. The result is REMEDIATED or REMEDY FAILED.
  5. At the rule's next interval the validation runs again on its own and gives the real verdict.

This happens every time the rule fails, on every failing host, as long as the rule has remediation.

The server does not send remediation when the rule is no longer assigned to the host, when the rule is inactive, or when a host in probation would need secret parameters.

Remediation is off in library rules

Rules from the LEA rule library contain a reviewed fix, but it is switched off: it is stored under the name _remedy instead of remedy. The server only runs remedy.

To read a library rule's disabled fix:

  1. Open the rule page → Definition tab.
  2. In Raw definition, click Show JSON.
  3. Look for "_remedy". Each entry has a command and the system it is for. An empty list [] means the library has no safe automatic fix for this rule.

Note: The rule editor does not show _remedy, and saving a rule in the editor does not keep it. Copy the commands you need before you edit the rule.

Turn on remediation safely

This procedure enables a fix for a test group first, then for everyone.

  1. Read the fix. Open the rule → Definition → Show JSON and copy the _remedy commands (or write your own; see Write a safe remedy). Make sure you understand every command.
  2. Open the rule and click Edit.
  3. Open Remediations and click Add remediation.
  4. Paste the command. Set Systems (and Target Distributions if the fix is for some distributions only) to the same values as the matching validation.
  5. Repeat for each system the rule covers.
  6. Click Save rule. The rollout dialog opens.
  7. Choose New hosts only and click Apply. All hosts that run the rule now stay on the version without remediation.
  8. Open your test group → Held rules tab → Update all, and confirm. Only the test group gets the version with remediation.
  9. Wait for the rule to run on the test hosts. Open the result of a failing host. You should see a Last remedy panel with REMEDIATED, and after the next interval the verdict PASS. See See what remediation did.
  10. When the test hosts look right, promote everyone: open the rule → Versions tab → Held hosts → Update all.

If something goes wrong in step 9, turn remediation off (see Turn remediation off). The other hosts never got it.

Write a safe remedy

A remedy runs automatically on many hosts, so it must be:

  • Safe on every host the rule targets. Limit it with Systems and Target Distributions.
  • Idempotent. Running it twice must give the same result as running it once. Set a value; do not append it.
  • Non-interactive. It must never wait for input.
  • Quiet. Send the fix's own output to /dev/null. The agent runs the validation again right after the fix, and the server checks the fix's output together with the validation output. Extra output can make the check fail, or pass by mistake.

Example. A rule reads cat /proc/sys/kernel/kptr_restrict and expects 1 or 2. A safe remedy is:

sysctl -w kernel.kptr_restrict=1 >/dev/null

It sets the value and hides its own output. The agent then runs the validation cat /proc/sys/kernel/kptr_restrict, the server sees 1 and reports REMEDIATED. If setting the value fails, the value is still 0 and the server reports REMEDY FAILED.

Note: Fixes from older rule library releases end with && cat (or ; reg query, ; auditpol /get) of the value they set. The value is then printed twice, and an expectation anchored with ^ and $ (such as ^(?:1|2)$) reports REMEDY FAILED even when the fix worked. Current library fixes are quiet. If you copied an older fix, remove that last part.

Good to know:

  • Remedies run as the LEA agent's user, like validation commands. That user has no administrator rights: lea on Linux and macOS, the service account NT SERVICE\LEAAgent on Windows. A fix that needs administrator rights fails unless you allow that user to run exactly that command. On Linux, add one line per command to a file in /etc/sudoers.d/; a rule that needs this says which line in its description.
  • A setting made at runtime (such as sysctl -w) may be lost at reboot. The rule then fails again and the remedy runs again. To make it permanent, also write the setting to its configuration file.
  • Test the commands on one host first, for example with the host Terminal tab for read commands, before you put them in a rule.

Named actions for functions

Under Remediations → Actions a rule can hold named lists of commands, for example install_cert. Actions never run on their own. A server-side function attached to the rule can pick one (Then on the host → Let the function pick an action). The host then runs the chosen action and right after it the rule's validations, so the result is checked at once.

To add one: click Add action, enter an Action name (lowercase letters, digits and _), and add its commands with Systems. See Functions.

See what remediation did

WhereWhat you see
Result page → Latest result → Last remedyWhen the remedy last ran, its return code, REMEDIATED or REMEDY FAILED, the failed expectation, and remedy stdout / remedy stderr. "via" and a function name when a function released it.
Result page → Latest result → Last actionOnly for rules with functions: the last action a function chose, with its output and the verdict of the validation that ran right after it.
Result page → Track recordThe number of Remedies (and Actions, when a function chose any), and a Pending remedy when one is waiting for a function.
Result page → Run historySet Type to Remedies to list every remedy run. The Type column shows remedy, or remedy · via and the function name. Actions are listed under Actions, not as remedies.
Rule preview (click a rule anywhere)Remediation · 30 days: Remedy success (share and count; actions chosen by functions are not counted), Functions ok, and p95 (95% of runs finished within this time).
Rule page → OverviewRemedy: automatic when the rule has remediation, otherwise none.

See Read results and fix failures.

Turn remediation off

For one rule:

  1. Open the rule and click Edit.
  2. Open Remediations and remove every command with its remove icon.
  3. Click Save rule, choose All hosts now and click Apply.

In an emergency, the fastest way is to stop the rule: edit it, turn Active off in General and click Save rule. This applies to all hosts immediately, but the rule also stops checking.

When to use a function instead

A remedy is a fixed command that runs on the host. Use a server-side function when:

  • the fix needs data the host does not have, such as a certificate signed by your internal certificate authority,
  • the fix needs a secret that must never be stored in a rule,
  • something must happen outside the host, such as opening a ticket or sending a message,
  • the fix must be computed for each host from its output,
  • you only want to record extra information next to the result.

A function can also prepare the data for the remedy (Run the remediation with the output) or choose one of the rule's actions. See Functions.

If something goes wrong

  • REMEDY FAILED every time: read remedy stderr on the result page. Common causes: the agent user lacks rights, the command does not exist on this distribution, or the remedy prints its own output that breaks the expectations (send it to /dev/null).
  • REMEDIATED, but the next check fails again: the fix does not last (for example lost at reboot or overwritten by configuration management). Make the fix permanent, or fix it in your configuration tool.
  • No Last remedy panel: the rule has no remediation, the host is held on an older version (check Held rules), or the rule has not failed since.
  • Remediation runs on hosts it should not: turn it off now (see Turn remediation off), then limit it with Systems and Target Distributions.

For more, see Troubleshooting.