Tryhackme | Intro to Logs | Walkthrough

In this room, you will learn the fundamentals of logging, data sources, collection methods, and principles.

Cecilie_Arcurs on Getty Images

Task 1: Introduction

The focus of the task is: How we can identify malicious activities? What kind of evidence is generated when an intruder breaches a network? Why it is essential to recognize these indicators within our environment?

Highlights: Room Learning Objectives
This room covers how logs can be used to record an adversary’s actions, the tools and techniques needed to perform log analysis, and the significance of effectively collecting and analyzing logs.

• Understand the importance of logs as a historical activity record for identifying and mitigating potential threats
• Explore various types of logs, logging mechanisms, and collection methods across multiple platforms
• Gain hands-on experience detecting and defeating adversaries through log analysis

Question:

I’m ready to learn more about logs.

Answer: No Answer Needed (Just click on the answer box)

Task 2: Expanding Perspectives: Logs as Evidence of Historical Activity

Working with Logs: Scenario: Room Highlights

Scenario: A web server of SwiftSpend Financial is constantly bombarded with scans from an adversary. As a systems administrator of this organisation tasked to address this predicament, you must identify what the adversary is doing by configuring logging and analysing collected logs.

IMPORTANT: The user Damianhall has limited sudo privileges. Issue the command sudo -l to check what commands can be run by this user. These limited commands are all that are needed to complete the subsequent tasks. (The user name and the command is important) take note of that.

Now start the machine to begin the task

If the VM is not visible, use the blue Show Split View button at the top-left of the page. Alternatively, using the credentials below, you can connect to the VM via RDP or SSH.

Username: damianhall

Password: Logs321!

IPMACHINE_IP: (once you start the machine, an IP address will be generated for you)

IMPORTANT: Escalation of Privileges is NOT necessary to answer the questions in this room. Meaning it is a very easy task.

Now go ahead and read through the rest of the task before going to the answers.

Question 1:

What is the name of your colleague who left a note on your Desktop?

No doubt there is an alternative way to do it. But once my machine was up, I saw a note.txt file. And when I saw the question I immediately clicked on the .txt file. The file contains a note to owner Damain, you can use the note to answer the rest of the questions.

Answer : Check Line 17 for the question to question 1.

Question 2:

What is the full path to the suggested log file for initial investigation?

Answer: Check the answer on line 7 of the note. The answer begins with ‘/var/log/gitlab…..

Task 3: Types, Formats, and Standards

Highlights: Common Log Types

• Application Logs: Messages about specific applications, including status, errors, warnings, etc.
• Audit Logs: Activities related to operational procedures crucial for regulatory compliance.
• Security Logs: Security events such as logins, permissions changes, firewall activity, etc.
• Server Logs: Various logs a server generates, including system, event, error, and access logs.
• System Logs: Kernel activities, system errors, boot sequences, and hardware status.
• Network Logs: Network traffic, connections, and other network-related events.
• Database Logs: Activities within a database system, such as queries and updates.
• Web Server Logs: Requests processed by a web server, including URLs, response codes, etc.

Log Formats

• Semi-structured Logs: Combines structured and unstructured data. Examples include Syslog and Windows Event Log.
• Structured Logs: Follows a strict format, easy for analysis. Examples are CSV, JSON, W3C Extended Log Format, and XML.
• Unstructured Logs: Free-text logs, rich but harder to parse. Examples include NCSA Common and Combined Log Formats.

Log Standards

• Common Event Expression (CEE): This standard, developed by MITRE, provides a common structure for log data, making it easier to generate, transmit, store, and analyse logs.
• OWASP Logging Cheat Sheet: A guide for developers on building application logging mechanisms, especially related to security logging.
• Syslog Protocol: Syslog is a standard for message logging, allowing separation of the software that generates messages from the system that stores them and the software that reports and analyses them.
• NIST Special Publication 800–92: This publication guides computer security log management.
• Azure Monitor Logs: Guidelines for log monitoring on Microsoft Azure.
• Google Cloud Logging: Guidelines for logging on the Google Cloud Platform (GCP).
• Oracle Cloud Infrastructure Logging: Guidelines for logging on the Oracle Cloud Infrastructure (OCI).
• Virginia Tech — Standard for Information Technology Logging: Sample log review and compliance guideline.

Question 1:

Based on the list of log types in this task, what log type is used by the log file specified in the note from Task 2?

Go back and look at the different types of logs, you will notice that server logs examples include access logs. But wait, it is not just an access log, you need to search for a suspicious web browser activity, and what type of log have requests processed by a web server?

Answer: Web…

Question 2:

Based on the list of log formats in this task, what log format is used by the log file specified in the note from Task 2?

Unstructured log Formats are mostly associated with web and HTTP servers. But the combined is used by default and since we are not told the type of log server, we go with the default.

Answer: Combined

Task 4: Collection, Management, and Centralisation

Highlights: Log Collection

• Focuses on the aggregation of logs from various sources.
• Maintaining accurate system time using Network Time Protocol (NTP) is essential.
• Steps:

1. Identify log sources.
2. Choose a log collector tool.
3. Set collection parameters, ensuring time synchronization with NTP.
4. Test the log collection process.

• Note: While using VMs without internet access, NTP might not be available.

Log Management

• Concerned with the secure, organized storage of logs and their efficient retrieval.
• Key steps:

1. Securely store logs, considering retention and access.
2. Organize logs by source, type, etc.
3. Backup logs regularly.
4. Periodically review stored logs.

Log Centralisation

• Centralises logs for faster access, in-depth analysis, and quick incident response.
• Steps:

1. Choose a system like Elastic Stack or Splunk for log centralization.
2. Connect all log sources to the system.
3. Monitor logs in real-time and set up alerts.
4. Integrate the central system with existing incident management tools.

Practical Activity: Log Collection with Rsyslog (Do not try to skip this part, you can't answer the questions without having done this)

This activity's aim is to configure rsyslog to log all sshd messages to a specific file, such as /var/log/websrv-02/rsyslog_sshd.log.

What you Need to do:

1. Open a Terminal.
2. Check if rsyslog is installed by running the command: sudo systemctl status rsyslog
3. Then use a text editor to create the following configuration file: gedit /etc/rsyslog.d/98-websrv-02-sshd.conf, nano /etc/rsyslog.d/98-websrv-02-sshd.conf, vi /etc/rsyslog.d/98-websrv-02-sshd.conf, or vim /etc/rsyslog.d/98-websrv-02-sshd.conf

(use one of the options not everything. I like to work with nano so i created a nano file)

4. Add the following lines in /etc/rsyslog.d/98-websrv-02-sshd.conf to direct the sshd messages to the specific log file just like the picture below:

5. Save and Close the Configuration File.

6. Apply the changes by restarting the rsyslog with the command: sudo systemctl restart rsyslog

Now to the Questions:

Question 1:

After configuring rsyslog for sshd, what username repeatedly appears in the sshd logs at /var/log/websrv-02/rsyslog_sshd.log, indicating failed login attempts or brute forcing?

After you have configured the file, you can check my commands if you are confused, to answer question 1, you need to read the file. use cat rsyslog_ssh.log command to read the file. There is an invalid user. what is the person’s name?

Answer: …(Tip) The Invalid user name

Question 2:

What is the IP address of SIEM-02 based on the rsyslog configuration file /etc/rsyslog.d/99-websrv-02-cron.conf, which is used to monitor cron messages?

Use the Cat command to read the file the @@ I added was to force it to give me an IP address.

Answer: @10.10.10.101

Question 3:

Based on the generated logs in /var/log/websrv-02/rsyslog_cron.log, what command is being executed by the root user?

Check CMD which is an abbreviated form of command. everything inside the bracket is your answer.

Answer: /bin/bash -c “/bin/bash -i >&…..(Get everything in the bracket)

Task 5: Storage, Retention, and Deletion

Highlights: Log Storage

The choice of storage location typically depends on multiple factors:

• Security Requirements
• Accessibility Needs
• Storage Capacity
• Cost Considerations
• Compliance Regulations
• Retention Policies
• Disaster Recovery Plans

Log Retention

• Hot Storage: Logs from the past 3–6 months that are most accessible.
• Warm Storage: Logs from six months to 2 years, acting as a data lake, easily accessible but not as immediate as Hot storage.
• Cold Storage: Archived or compressed logs from 2–5 years.

Log Deletion

Log deletion helps to:

• Maintain a manageable size of logs for analysis.
• Comply with privacy regulations, such as GDPR, which require unnecessary data to be deleted.
• Keep storage costs in balance.

Practical Activity: Log Management with logrotate

Create a Configuration File: sudo gedit /etc/logrotate.d/98-websrv-02_sshd.conf, sudo nano /etc/logrotate.d/98-websrv-02_sshd.conf, sudo vi /etc/logrotate.d/98-websrv-02_sshd.conf, or sudo vim /etc/logrotate.d/98-websrv-02_sshd.conf

• Create a Configuration File: sudo gedit /etc/logrotate.d/98-websrv-02_sshd.conf, sudo nano /etc/logrotate.d/98-websrv-02_sshd.conf, sudo vi /etc/logrotate.d/98-websrv-02_sshd.conf, or sudo vim /etc/logrotate.d/98-websrv-02_sshd.conf
• A script has been written for you to use and define log settings. Just copy it and past in the nano file you created
• Then Save and Close the file.
• Use the following command to do a Manual Execution: sudo logrotate -f /etc/logrotate.d/98-websrv-02_sshd.conf

Question 1:

Based on the logrotate configuration /etc/logrotate.d/99-websrv-02_cron.conf, how many versions of old compressed log file copies will be kept?

Read the file you created to find the number of compression files copies kept.

Answer: 24

Question 2:

Based on the logrotate configuration /etc/logrotate.d/99-websrv-02_cron.conf, what is the log rotation frequency?

The first thing we see in the logs is the frequency the log is set to rotate. which is hourly

Answer: hourly

Task 6: Hands-on Exercise: Log analysis process, tools, and techniques

Highlights:

Log Analysis Process

Log Analysis Tools

Unparsed Raw Log Files

Parsed and Consolidated Log File

Question 1:

Upon accessing the log viewer URL for unparsed raw log files, what error does “/var/log/websrv-02/rsyslog_cron.log” show when selecting the different filters? To answer this question you have to click on the log viewer tool link.

Then add /var/log/websrv-02/rsyslog_cron.log filter

| Answer: no date field

Question 2:

What is the process of standardising parsed data into a more easily readable and query-able format?

| Answer: Normalisation

Question 3:
What is the process of consolidating normalised logs to enhance the analysis of activities related to a specific IP address?

Answer: enrichment

Task 7: Conclusion

Congratulations! You’ve completed the Intro to Logs room.

In summary, we were able to learn and perform the following:

• The significance of logs as records of past activities; is essential for pinpointing and tackling threats.
• Delve into an array of logs, their creation techniques, and the methods of gathering them from diverse systems.
• Review the results from analysing logs in the realms of detection engineering and incident handling.
• Acquire practical skills in identifying and countering adversaries via log analysis.

This was a great experience and I enjoyed every bit of it as I hope you did. And if you did use the login details please let me know where and how you did it.