File Permission Problems: Reducing Security Vulnerability around Splunk Administration

File Permission Problems: Reducing Security Vulnerability around Splunk Administration

File system permissions in Linux, the suggested operating system for Splunk, can be worrisome for Splunk administrators.  Often times, a Splunk Administrator needs correct permissions to read system logs; however, by default in Linux, these permissions are not available, or only available to the one Linux super user account.

To work around this, a common configuration in the field for Linux based systems is where root is configured as the user to run the Splunk system.  This configuration alleviates the complaints that the Splunk non-root user does not have the proper permissions to access and view the data.  Naturally, running the program as the root user will allow access to the files that are being read.

The drawback in doing this, is that running Splunk as the root user creates a security vulnerability for these systems. Due to this, best practices from Splunk Professional Services suggest not running Splunk as the root user. Surprisingly, a large number of Splunk consulting engagements that I have been on, have ignored this security risk. This article suggests ways to reduce the Splunk security risks associated with root access, stay in bounds on Splunk best practices, while simultaneously providing Splunk Administrators the access to the data they need access to.

The traditional fix that does not involve running the Splunk software as root is to chmod the files or directories with permissions that can be set for the owner, group and everyone.  It is not uncommon to manipulate the group ownership of a file so that the non-root Splunk user can view and subsequently index its contents.  While that solution works, there is a large risk of causing other issues on the system.

What next?

Thankfully, the getfacl and setfacl commands are available and allow directory/file access control for file access.  The command runs as root or as an authorized sudo user.  Individual users and/or groups can be given read, write and/or execute permissions.

An example command would be:

setfacl -m u:splunk:rx <full path and filename>

File permissions fix

The setfacl command however only solves the issue if you are aware which files needed to have their permissions modified before deploying your configuration.  To fix the file access permissions after the fact can be done with a search of the splunk.log using Splunk for the host in question or a script can be run like the one below.

The script below is a great example that will help to alleviate the permissions concerns.  The script works by reading the splunkd.log and picking out the files that have been reported with insufficient permissions.  The script will then process the command above with the spunk user as the intended recipient of the new file permissions.  Bundled into the script are a couple of options that allow setting:

  • Splunk location (Enterprise or Universal Forwarder)
    • SUF=”/opt/splunkforwarder/var/log/splunk”
    • #
    • SDEF=”/opt/splunk/var/log/splunk”
  • Debug (logged output)
    • DEBUG=1
  • Test Run (additional verification step if needed)
    • TEST=1

The ideal location to run the script is from the /tmp directory and as the root user account. Logic has been added into the script where each file is tested for existing permissions, if the permissions already exist a message is logged with that information.

The Script:

#! /bin/bash

#

#  This script will modify the acl rights of files retrieved from

#  the /opt/splunk/var/log/splunk/splunkd.log

#  that need to allow the [splunk] user to have read and execute acess.

#

#

#

####################### script initializaiton ##

#

# To run on splunkforwarder (UF) or a custom path

# set set value to the [splunkd.log] path for this installation.

# If path exist then log will be read from that location.

#################################################

#

SUF=”/opt/splunkforwarder/var/log/splunk”

#

SDEF=”/opt/splunk/var/log/splunk”

if [ -d $SUF ]

then

  grep permission $SUF/splunkd.log | awk -F “‘” ‘{print $2}’ > /tmp/splperm

else

  grep permission $SDEF/splunkd.log | awk -F “‘” ‘{print $2}’ > /tmp/splperm

fi

#

touch /tmp/aclresults

touch /tmp/acltest

FILES=”/tmp/splperm”

rFILES=”/tmp/aclresults”

aclEXIST=”/tmp/acltest”

INPUT=`cat $FILES`

###############################################

 

######################## script options #######

#

# To see full before and after results per file set value to 1

DEBUG=1

# To initialize a test run and verify correct files set value to 1

TEST=1

###############################################

 

######################## main script ##########

echo `date` > $rFILES

for f in $INPUT

do

        getfacl $f | grep splunk > $aclEXIST

        if [ ! -s $aclEXIST ]

        then

          echo `date` ” Processing splunk user rights for $f” >> $rFILES

          if [ $DEBUG -eq 1 ]

          then

            echo “Before:” >> $rFILES

            getfacl $f >> $rFILES

          fi

          if [ $TEST -eq 0 ]

          then

            echo “This is not a test run for $f” >> $rFILES

            setfacl -m u:splunk:rx  $f

          else

            echo “This is a test run for $f” >> $rFILES

          fi

          if [ $DEBUG -eq 1 ]

          then

            echo “After:” >> $rFILES

            getfacl $f >> $rFILES

          fi

        else

          echo `date` “Splunk user rights for $f not needed” >> $rFILES

        fi

done

################### main script end ########

Review

File permissions can be a big problem when trying to index data into Splunk.  The steps to mediate the issues can range from simple to difficult. Using chmod may open other issues with the system and its overall security.  Using the getfacl and setfacl commands expand on the chmod command in Linux allowing per user access, while also allowing greater control over file permissions and access. These commands make them the preferred method of balancing Splunk Administration and system controls.

Using the script above automates the repair process that will allow files with permissions errors to be indexed into the Splunk environment. The script goes a little further by providing output logs that can be audited if the script is to be ran on a schedule or on-demand.

As always, Splunk Answers can the go-to source for help and further information on this topic can be found on Splunk Answers (https://answers.splunk.com/index.html).  Splunk professional services firms such as Aditum are also always here to help Splunk customers with these and many other issues, to make sure that your Splunk instance is optimized and driving significant business value for your organization.

This article was authored by Cedric Milan, a senior Splunk Professional Services consultant with Aditum. Cedric is a Splunk Certified Consultant II, the highest level of certification in the Splunk domain. Aditum is a leading Splunk business partner, assisting customers with software acquisition and Splunk consulting services.

About Aditum

Aditum (Latin: “to access”) is a professional services firm specializing in next-generation Analytics solutions based upon the Splunk platform. The company is quickly building one of North America’s largest and most accomplished Splunk consulting teams with extensive experience across all of Splunk’s major use cases including IT Ops, Security, DevOps, Business Analytics and IoT.

Aditum’s Splunk Certified Architects and Splunk Certified Consultants drive client success with initial Splunk deployments, environment upgrades/scaling, building of dashboards/searches/reports and Splunk health checks. Aditum also has a team of accomplished Splunk Developers that focus on the development of Splunk apps and technical add-ons.