Overview of Log Management and Monitoring
Log management and monitoring are essential parts of modern IT infrastructure and cybersecurity. They involve collecting, storing, analyzing, and monitoring log data from various systems, applications, and devices within an organization. Effective log management helps to identify and troubleshoot issues, ensures compliance with regulations, and enhances security by detecting anomalies and potential threats.
By implementing strong log management and monitoring practices, organizations can improve operational efficiency, boost security, and ensure regulatory compliance.
In this guide, we will send our logs to Splunk Cloud using the Splunk Universal Forwarder on various operating systems.
Splunk Cloud Preparation
We need to prepare our Splunk Cloud to receive and index our logs appropriately. To achieve this, we must configure the following:
Index
Navigate to
<your_organization>.
splunkcloud.com
>Settings
>Index
>HTTP
>New Index
.Enter an index name and configure it as needed. Click on Save.
HEC (HTTP Event Collector)
Go to
<your_organization>.
splunkcloud.com
>Settings
>Data inputs
>HTTP Event Collector
>Add new
>New Token
.Set a name for the HEC. You can use the same name as your index for consistency. Click on Next.
In the Default Index field, select the index you created. Click on Review.
Click on Submit and copy your Token Value.
With these configurations in place, our Splunk Cloud is ready to receive logs from various hosts using Splunk Universal Forwarders (agents).
Installation & Configuration for Windows
You can follow these steps to install and configure the Splunk Universal Forwarder on Windows systems.
Download the Splunk Universal Forwarder installer from here. (You will need to create an account to be able to download the installer)
Install the forwarder by clicking "Next" through the prompts.
When asked for credentials, set them and save them somewhere safe.
When asked for monitoring options, select what you need to monitor.
When asked about the receiving server, leave it blank. We will configure it later.
Download your .spl file containing the credentials to connect to your Splunk Cloud.
It is usually found at
<your_organization>.
splunkcloud.com
>Apps
>Universal Forwarder
>Download Universal Forwarder Credentials
Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/n..
Open CMD.
Navigate to Splunk’s installation folder with the command:
cd C:\Program Files\SplunkUniversalForwarder\bin
Install credentials:
.\splunk.exe install app %HOMEPATH%\Downloads\splunkclouduf.spl
Restart the service:
.\splunk.exe restart
Send data to SplunkCloud:
.\splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
Navigate to the directory containing your
inputs.conf
andoutputs.conf
files:cd C:\Program Files\SplunkUniversalForwarder\system\local
Configure your
inputs.conf
file as needed. Here you have an example:[WinEventLog://Application] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest index=<name_of_index> [WinEventLog://Security] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest index=<name_of_index> [WinEventLog://System] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest index=<name_of_index> [WinEventLog://ForwardedEvents] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest index=<name_of_index> [WinEventLog://Setup] checkpointInterval = 5 current_only = 0 disabled = 0 start_from = oldest index=<name_of_index> [admon://NearestDC] monitorSubtree = 1 index=<name_of_index>
Configure your
outputs.conf
file with the following content:[httpout] httpEventCollectorToken = <your_HEC_token> uri = https://http-inputs-<your_organization>.splunkcloud.com:443
Set the
SPLUNK_HOME
environment variable with this command:set SPLUNK_HOME=C:\Program Files\SplunkUniversalForwarder\opt
Restart the service:
cd C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe restart
Find your computer name using this command:
hostname
On SplunkCloud, search for this SplunkQL query:
index="main" host="<write_your_hostname>"
If you see events, congratulations! You were successful!
PS1: If you need to add a new monitor on a file or folder, use this command:
.\splunk add monitor "<path to file or folder>"
(the path goes between quotes!)
PS2: To change the index where you send logs, go to:
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local
and add a line in each section of the inputs.conf
file like this:
index=<name_of_index>
Installation & Configuration for RedHat-based distributions
You can follow these steps to install and configure the Splunk Universal Forwarder on a RedHat-based distribution such as CentOS, Fedora, Oracle Linux, etc.
Download the Splunk Universal Forwarder installer from here. (You will need to create an account to download the installer)
Install the forwarder:
rpm -i <installer_name>.rpm
Download your .spl file containing the credentials to connect to your Splunk Cloud.
It is usually found at
<your_organization>.
splunkcloud.com
>Apps
>Universal Forwarder
>Download Universal Forwarder Credentials
Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/n..
You can upload your .spl to a file storage so you can download it in step 6.
Go to the path where the Splunk bin is located:
/opt/splunkforwarder/bin/splunk start
Enter your user and password. It's important to save them somewhere safe because we'll need them to configure things in the future.
Download
splunkclouduf.spl
:curl -O https://<url_to_your_spl_file>
Install it:
/opt/splunkforwarder/bin/splunk install app splunkclouduf.spl
Send data to Splunk Cloud:
./splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
Navigate to the directory containing your
inputs.conf
andoutputs.conf
files:cd /opt/splunkforwarder/etc/system/local
Configure your
inputs.conf
file as needed. Here you have an example:# Monitor system logs for authentication and authorization events [monitor:///var/log/secure] disabled = false index = <name_of_index> sourcetype = linux_secure # Monitor system logs for general security events [monitor:///var/log/messages] disabled = false index = <name_of_index> sourcetype = linux_syslog # Monitor Apache access and error logs [monitor:///var/log/httpd/access_log] disabled = false index = <name_of_index> sourcetype = apache_access [monitor:///var/log/httpd/error_log] disabled = false index = <name_of_index> sourcetype = apache_error # Monitor SSH logs for login attempts [monitor:///var/log/secure] disabled = false index = <name_of_index> sourcetype = sshd # Monitor sudo commands executed by users [monitor:///var/log/secure] disabled = false index = <name_of_index> sourcetype = sudo # Monitor firewall logs (assuming firewalld is used) [monitor:///var/log/firewalld] disabled = false index = <name_of_index> sourcetype = firewalld # Monitor audit logs (if available) [monitor:///var/log/audit/audit.log] disabled = false index = <name_of_index> sourcetype = linux_audit # Monitor file integrity using auditd (if available) [monitor:///var/log/audit/auditd.log] disabled = false index = <name_of_index> sourcetype = auditd # Monitor for changes to critical system files [monitor:///etc/passwd] disabled = false index = <name_of_index> sourcetype = linux_config # Monitor for changes to critical system binaries [monitor:///bin] disabled = false index = <name_of_index> sourcetype = linux_config # Monitor for changes to critical system configuration files [monitor:///etc] disabled = false index = <name_of_index> sourcetype = linux_config
Configure your
outputs.conf
file with the following command:echo "[httpout] httpEventCollectorToken = <your_HEC_token> uri = https://http-inputs-<your_organization>.splunkcloud.com:443" > outputs.conf
Set the
SPLUNK_HOME
environment variable:export SPLUNK_HOME=/opt/splunkforwarder
Restart the Splunk service to apply the new configuration:
/opt/splunkforwarder/bin/splunk restart
PS1: To add a new monitor on a file or folder, use the following command:
/opt/splunkforwarder/bin/splunk add monitor "<path to file or folder>"
(The path should be enclosed in quotes)
PS2: To change the index where you send logs, navigate to the directory:
cd /opt/splunkforwarder/etc/system/local
and add a line in each section of the inputs.conf
file like this:
index=<name_of_index>
Installation & Configuration for Ubuntu
You can follow these steps to install and configure the Splunk Universal Forwarder on a Debian-based distribution such as Ubuntu, Debian, etc.
Download the Splunk Universal Forwarder installer from here. (You will need to create an account to download the installer)
Install the forwarder:
dpkg -i <installer_name>.deb
Download your .spl file containing the credentials to connect to your Splunk Cloud.
It is usually found at
<your_organization>.
splunkcloud.com
>Apps
>Universal Forwarder
>Download Universal Forwarder Credentials
Alternatively, you can find it here: https://<your_organization>.splunkcloud.com/splunkd/__raw/servicesNS/n..
You can upload your .spl to a file storage so you can download it in step 6.
Go to the path where the Splunk bin is located:
/opt/splunkforwarder/bin/splunk start
Enter your user and password. It's important to save them somewhere safe because we'll need them to configure things in the future.
Download
splunkclouduf.spl
:curl -O https://<url_to_your_spl_file>
Install it:
/opt/splunkforwarder/bin/splunk install app splunkclouduf.spl
Send data to Splunk Cloud:
./splunk add forward-server http-inputs-<your_organization>.splunkcloud.com:443
Navigate to the directory containing your
inputs.conf
andoutputs.conf
files:cd /opt/splunkforwarder/etc/system/local
Configure your
inputs.conf
file as needed. Here you have an example:#fix bug in ubuntu related to: "Events from tracker.log have not been seen for the last 90 seconds, which is more than the yellow threshold (45 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked." [health_reporter] aggregate_ingestion_latency_health = 0 [feature:ingestion_latency] alert.disabled = 1 disabled = 1 # Monitor system logs for authentication and authorization events [monitor:///var/log/auth.log] disabled = false index = <name_of_index> sourcetype = linux_secure # Monitor system logs for general security events [monitor:///var/log/syslog] disabled = false index = <name_of_index> sourcetype = linux_syslog # Monitor Apache access and error logs [monitor:///var/log/apache2/access.log] disabled = false index = <name_of_index> sourcetype = apache_access [monitor:///var/log/apache2/error.log] disabled = false index = <name_of_index> sourcetype = apache_error # Monitor SSH logs for login attempts [monitor:///var/log/auth.log] disabled = false index = <name_of_index> sourcetype = sshd # Monitor sudo commands executed by users [monitor:///var/log/auth.log] disabled = false index = <name_of_index> sourcetype = sudo # Monitor UFW firewall logs (assuming UFW is used) [monitor:///var/log/ufw.log] disabled = false index = <name_of_index> sourcetype = ufw # Monitor audit logs (if available) [monitor:///var/log/audit/audit.log] disabled = false index = <name_of_index> sourcetype = linux_audit # Monitor file integrity using auditd (if available) [monitor:///var/log/audit/auditd.log] disabled = false index = <name_of_index> sourcetype = auditd # Monitor for changes to critical system files [monitor:///etc/passwd] disabled = false index = <name_of_index> sourcetype = linux_config # Monitor for changes to critical system binaries [monitor:///bin] disabled = false index = <name_of_index> sourcetype = linux_config # Monitor for changes to critical system configuration files [monitor:///etc] disabled = false index = <name_of_index> sourcetype = linux_config
Configure your
outputs.conf
file with the following command:echo "[httpout] httpEventCollectorToken = <your_HEC_token> uri = https://http-inputs-<your_organization>.splunkcloud.com:443" > outputs.conf
Set the
SPLUNK_HOME
environment variable:export SPLUNK_HOME=/opt/splunkforwarder
Restart the Splunk service to apply the new configuration:
/opt/splunkforwarder/bin/splunk restart
PS1: To add a new monitor on a file or folder, use the following command:
/opt/splunkforwarder/bin/splunk add monitor "<path to file or folder>"
(The path should be enclosed in quotes)
PS2: To change the index where you send logs, navigate to the directory:
cd /opt/splunkforwarder/etc/system/local
and add a line in each section of the inputs.conf
file like this:
index=<name_of_index>
Conclusion: Streamline Your Log Management with Splunk Cloud
By following this guide, you can efficiently send logs to Splunk Cloud from various operating systems, ensuring that your log data is centralized, easily accessible, and ready for analysis. Proper log management and monitoring are crucial for maintaining the health, security, and compliance of your IT infrastructure. With Splunk Cloud and the Splunk Universal Forwarder, you can streamline these processes and gain valuable insights from your log data. If you have any questions or need further assistance, feel free to contact us. We'll be happy to help. Happy logging!