Snorby script progress, and a rundown of how to perform the installation.

Hello Autosnort users,

Still working on the Snorby release for Autosnort, and even then, It's probably going to be a staggered release -- first Ubuntu, then Debian, and Finally moving to CentOS. I imagine that based on some of the issues I have been experiencing, Getting Snorby to play with with SELinux is going to be a lot of fun.

I managed to get it to work with Aanval, snortreport and BASE, so I can do it again.

Anyhow, for those of you who can't wait for me to release a script, here's a basic run-down of the commands I'm running in Ubuntu 12.04 that  I've used that work. this assumes you've ran autosnort up to the point where web interface installation takes place.

Don't be alarmed when it says you're downloading 400m of packages. that's about right:

apt-get install libyaml-dev git-core imagemagick libmagickwand-dev wkhtmltopdf libssl-dev libxslt1-dev libsqlite3-dev libmysql++-dev libcurl4-openssl-dev apache2-prefork-dev default-jre-headless

this installs rvm. rvm is a nice little program that can be used to automatically install ruby:

\curl -\#L https://get.rvm.io | sudo bash -s stable
/usr/local/rvm/bin/rvm autolibs enable
source /etc/profile.d/rvm.sh

this is a cute little hack that I've used for installing the latest stable version of snort, DAQ, and determine what version of snort rules are available via snort.org. From what I gather ruby updates frequently. very, very frequently. This string of commands tells us the latest ruby 1.9.x release available for installation. Why not install ruby 2.0.0? because snorby doesn't support it yet. Submitted an issue to snorby's github. We'll see where that takes us:

wget http://ruby-lang.org/en/index.html -O /tmp/index.html
rubyver=`cat /tmp/index.html | grep -e "ruby-" | head -4 | grep released | grep -v Continue | cut -d"/" -f8 | cut -d" " -f2 | tail -1`
rvm install ruby-$rubyver

After all the packages you need to install, there's also a boatload of gems that need installation. and oh, we haven't pulled down the snorby web UI yet:

gem install thor i18n bundler tzinfo builder memcache-client rack rack-test rack-mount rails rake rubygems-update erubis mail text-format sqlite3 daemon_controller
gem install passenger
update_rubygems
cd /var/www/
git clone http://github.com/Snorby/snorby.git
cd snorby/config

Make copies of database.yml.example and snorby_config.yml.example and rename them to database.yml and snorby_config.yml:

cp database.yml.example database.yml
cp snorby_config.yml.example snorby_config.yml

modify snorby_config.yml. this should be the only change that needs to be made:
sed -i 's/usr\/local\/bin/usr\/bin/' snorby_config.yml

modify database.yml the snorby section at the top, you MUST enter the root database user's password. this is necessary for the rake snorby:setup portion

Next, we have to compile the mod_passenger apache module. this will take a little bit of time.
passenger-install-apache2-module

If the passenger-install command above bombs, try running "gem install passenger --pre" and running the command again.

The passenger-install command will ask you to modify a couple of files. First we have to point apache to the passenger module:

echo "" >> /etc/apache2/apache2.conf
echo "# This stuff is to make Snorby work properly mod_passenger is required for snorby to work." >> /etc/apache2/apache2.conf
echo "" >> /etc/apache2/apache2.conf
echo "LoadModule passenger_module /usr/local/rvm/gems/ruby-$rubyver/gems/passenger-4.0.0.rc6/libout/apache2/mod_passenger.so" >> /etc/apache2/apache2.conf
echo "PassengerRoot /usr/local/rvm/gems/ruby-$rubyver/gems/passenger-4.0.0.rc6" >> /etc/apache2/apache2.conf
echo "PassengerDefaultRuby /usr/local/rvm/wrappers/ruby-$rubyver/ruby" >> /etc/apache2/apache2.conf

Next, we create a virtualhost for snorby and set the documentroot to snorby's public directory, then disable the default site. This effect means that if you navigate to http://[ip address] you automatically get the index page for snorby:

echo "<VirtualHost *:80>" >> /etc/apache2/sites-available/snorby
echo "     ServerName snorby.localhost" >> /etc/apache2/sites-available/snorby
echo "     # !!! Be sure to point DocumentRoot to 'public'!" >> /etc/apache2/sites-available/snorby
echo "     DocumentRoot /var/www/snorby/public" >> /etc/apache2/sites-available/snorby
echo "     <Directory /var/www/snorby/public>" >> /etc/apache2/sites-available/snorby
echo "          # This relaxes Apache security settings." >> /etc/apache2/sites-available/snorby
echo "          AllowOverride all" >> /etc/apache2/sites-available/snorby
echo "          # MultiViews must be turned off." >> /etc/apache2/sites-available/snorby
echo "          Options -MultiViews" >> /etc/apache2/sites-available/snorby
echo "     </Directory>" >> /etc/apache2/sites-available/snorby
echo "</VirtualHost>" >> /etc/apache2/sites-available/snorby





All this has just been setup so we can successfully run bundler and rake to actually install snorby:

cd /var/www/snorby

bundle install --deployment

note: if the bundle install command fails or gives you an error referencing to the gemlock file or a gem called psych shield, you may have to run "bundle install --no-deployment" then run "bundle install --deployment" for things to work right.

finally, the moment you've been waiting for...:

rake snorby:setup

note: do not try to create the snorby database prior to running rake snorby:setup. If the database already exists, rake will bomb and exit.

At this point, provided rake snorby:setup ran, you should have a web interface that you can log into. default creds for snorby are: snorby@snorby.org with a password of snorby.

If you reboot your sensor, the worker jobs and the delayed_run jobs, processes necessary for populating the dashboard on the web UI do not automatically start with the system. You need to run these commands on reboot:

cd /var/www/snorby && ruby script/delayed_job start
cd /var/www/snorby && rails runner 'Snorby::Jobs::SensorCacheJob.new(false).perform; Snorby::Jobs::DailyCacheJob.new(false).perform'

What I am working on:
- Every time I've ran the installation, I've varied one thing (e.g. the first time through I tried installing with ruby 2.0.0, the second time around, I installed and found out daemon_controller was a needed gem that no install guide referenced, the third time around I tried installing by creating the snorby database and granting the snort database user access to the database. This is also when I encountered the problem with bundler complaining about the gemlock file and the psych shield gem and had to re-run bundler with the --no-deployment option.

I need to re-run the installation until I get absolutely zero errors. I want this fully repeatable with no errors.
I want to investigate the possibility of editing the database.yml post-installation: using the snort database user, and using mysql commands to grant the necessary privs to the snort mysql user instead of root. Having the root mysql user's privs in plaintext  in a file that's world readable by default sounds like a terrible idea to me.

possibly adding the delayed_job and SensorCacheJob commands to rc.local to ensure they are ran upon reboot.

Getting support for Snorby is my highest priority right now. After that is done, I have some efficiencies and changes to autosnort I'll talk about in another post. Until then,

Cheers from DA.

Short and Sweet

Autosnort users,

Two things to be aware of:

1) the pulledpork auto rule download functionality appears to be broken right now. This is due to a lack of forethought on my part on believing that there wouldn't be less than 30 days between two versions of snort being released. Working on fixing this as we speak.

2) I'm working pretty hard on automating installation of Snorby. Snorby is an ambitious project and a beautiful web interface, but the Documentation hasn't seen updates in over a year, and things that should be simple now have Ruby, rails and all the dependency hell that comes with it. It's coming together, slowly but surely.

Until next time.

Update: yesterday evening the snort 2.9.4.5 rules were made available to registered users. This sort of invalidates the fix I'm putting into autosnort, but I'm still going to do it to prevent this from happening again. Also still working on a snorby script.

Sometimes the old ways are the best ways (support for BASE and Syslog_Full output)

Hello AS/snort users,

Got two new features in this latest build of autosnort.

First and foremost is support for the BASE web interface. BASE is by and far one of the older and more well-known web interfaces for snort alerting. While the interface isn't going to win for prettiest interface, it is by and far very functional. Some of the cool things include the ability to download alerts as pcap and actual tshark/pcap parsed output for each alert.

This has been added as an installation option in the interface selection menu. The installation is painless and straight forward. After the system reboots, just point your browser to http://[your ip]/base.

The web page initial setup will check for all of the required pre-requisite settings on the first page. On the second page, it will ask you where adodb, a pre-requisite package, is installed. Autosnort and most linux distros install adodb to /usr/share/php/adodb (put that in for the path to adodb). It will also, of course ask you for database information and credentials:
database name: snort
database host: localhost
database port: 3306 (can leave blank if you'd like)
database username: snort
password: (the snort database user's password)

Unless you have an archive database configured, leave that part blank.
On the last page, it will ask if you want to set up auth to get into the web page to review events. That is entirely up to you.

After that, you should be all set to get events to the BASE web interface.

The next feature I have to announce is support for syslog output as another output interface choice. I've added a child script that configures barnyard2 to output to syslog_full format to port 514/udp. I have not FULLY tested this feature, but preliminary tests show that splunk picks up and parses the events with no issues whatsoever. A couple of troubleshooting recommendations:

-Ensure that the sensor's management interface, if it is firewalled, has port 514/udp open and allowed outbound.
-Conversely, ensure that your SIEM or syslog aggregator has port 514/udp open inbound to accept events -- I hosted splunk on a CentOS box and was puzzled why tcpdump says we were sending events out, but they were never making it to be indexed by splunk. I ran system-config-firewall-tui and added a rule for 514/udp inbound (and port 8000/tcp inbound for the actual web UI), and everything was fine.

The child scripts have been made available to Debian, Ubuntu and CentOS.

One last thing:

I've made some minor changes to the script at the recommendation of maintainer of the barnyard2 project. Instead of using sed to modify the copy of barnyard2.conf included with the barnyard2 source, I'm making the script generate a barnyard.conf on-the-fly.

- the script keeps a copy of the original barnyard2.conf in /usr/local/snort/etc/ named barnyard2.conf.origin
- the script generates a barnyard2.conf with only the directives necessary to run barnyard2 (e.g. where are the reference and classification config files, the gen/sid-msg.map files, input format to accept, output format to output to, etc.)
- if you choose to configure output to syslog format, the child script disables output to the database automatically.

As always, the scripts are up on github, along with updated READMEs. Previous versions are available in the Previous_Rel directory as well.

Happy Hunting..

DA_667

New AS release, New init scripts, bug fixes and more!

Hello AS users,

Got a lot to talk about today, so let's get down to business:

First and foremost I've finally gotten around to writing half-decent documentation for autosnort. I've pushed a pdf file to github featuring documentation with screenshots on how to install Autosnort via vmware player, including an interesting hack on how to get bridging to work on multiple interfaces on vmware player 5.

I chose CentOS as the vmware player operating system, but the instructions should be easy to follow for Debian Ubuntu or Backtrack. Any major differences were noted. All said and done it was about 23 pages of documentation with screenshots peppered in.

I focused on using VMware player this time around, and generally speaking the VMware player guidance can be used just as well for VMware Fusion or Workstation. I may do an alternate document for ESXi and/or Virtualbox in the near future.

I've also included general troubleshooting steps and things you *should* see after running autosnort.

Next in line is an update to the autosnort script for Debian, Ubuntu, and CentOS. Banyard2.13 BETA2 has changed how barnyard2 behaves. As a result of this, I had to change how arguments are passed to barnyard2 in Autosnort. The details are available for each operating system's readme files. Suffice to say, everything works fine now.

Finally, I've gotten around to testing and posting init scripts for Ubuntu and Debian to control snort and barnyard2. The Ubuntu and Debian init scripts support start, stop and restart functions and are pretty easy to implement. Implemtation details are also available via the Readme files for Ubuntu and Debian Autosnort scripts, but suffice to say, its incredibly easy.

I may release a modification to the init script that detects if you installed aanval and automatically start/stop the BPUs along with snort and barnyard2, but that will come a little later.

The documentation, new scripts, readmes and init scripts should all be available via github as of now.

Cheers,

DA

Redhat/CentOS init script!

Hello AS and regular snort users,

I wanted to announce that I have developed an init script compatible with CentOS/Redhat variants for autosnort installations (of course, this script can easily be adapted to other snort snort installations.

This init script can be used to replace /etc/rc.local as the primary method of starting up snort and barnyard2, and includes the added bonus of allow you to start/stop/restart snort and barnyard2 without requiring a reboot or sourcing /etc/rc.local if you need to make changes to snort or barnyard2. To add this script to CentOS 6.x perform the following tasks as root (or via sudo/root permissions):

    1. Copy the snortbarn script to /etc/init.d
    2. Edit the variables near the top of the script to suit your snort installation (the only variable that you should need to modify is the snort_iface variable if you installed snort/barnyard2 via autosnort)
    3. Make the snortbarn script is executable for the root user (chmod 700 snortbarn)
    4. Run chkconfig --add snortbarn
    5. Remove the entries for ifconfig, snort, and barnyard2 from /etc/rc.local (note: you may want to make a backup of the rc.local script in case you run into bugs/problems with the init script!)
    6. Kill your current snort/barnyard processes that ran from rc.local (killall snort && killall barnyard2)
    7. Run the command "service snortbarn start"
    8. check the process list to ensure that snort and barnyard2 are running after calling the init script. ( "ps -ef | grep snort" will return snort and barnyard2, if either/both processes are running. If only one process or the other is visible, something is wrong)

Troubleshooting steps: I'm not entirely sure why but there are CRLF/LF formatting problems with this script. If you get a bunch of errors stating that a file/command doesn't exist, try running dos2unix on the file to resolve the CRLF/LF errors.

If you install the init script and upon reboot find that only the snort process is running, it is because the init script for snortbarn ran BEFORE the init script for mysqld ran. To Determine when mysqld is configured to run it its runlevels, check /etc/init.d/mysqld. You'll want to pay attention to this line in particular:

# chkconfig: - 64 36
the first number, 64 indicates what number the rc startup script will get on startup. Linux rc scripts determine what services run or are killed on a particular run level. Every rc script as a K for Kill order, an S for Start order, followed by a number and the name of the symlinked script from /etc/init.d. RC scripts are read in numeric order. So if the rc script for snortbarn has an S number lower than 64, it will run before mysqld. snort will start up fine, but barnyard fails because it has no database to connect to.

To remedy this, you can modify the /etc/init.d/mysqld script to have a lower number than the snortbarn script in any of the /etc/rc[2-5].d directories, or modify the snortbarn script to have a higher number than the mysqld directory. This is a little confusing, so let's look at an example:

run this command: ls -al /etc/rc?.d/S*snortbarn

this command shows you each runlevel snortbarn is configured to start on.

now, run this command: ls -al /etc/rc?.d/S*mysqld

this command shows you the runlevels mysqld is set to run on. Don't worry about how many results you get.

If mysqld's number is higher than snortbarn's number, the mysqld process will not be running before snort and barnyard are configured to run. No database running means barnyard2 won't run. Let's say snortbarn had a number of 63, and mysqld has a number of 64. edit /etc/init.d/mysqld and change the chkconfig to something like this:

# chkconfig: - 62 36

save your changes and run chkconfig --add mysqld. This should fix the problem.

I've added the above documention to the other notes section of the CentOS readme. The snortbarn script is now available via github.

Special Note: Would like to thank Tactical FLEX/Aanval for hosting the initial version of this init script that I based this one off of. Would also like to thank Mike Miller for the initial idea of moving away from rc.local and building out a legitimate init script for autosnort.

Happy Snorting,

DA667

Hammering out Aanval

Hello Autosnort Users,

Today I would like to announce support for Tactical Flex's Aanval Web Console on CentOS and Debian platforms in addition to last weekend's Ubuntu release

I've implemented the same changes for the CentOS and Debian scripts that I've implemented on the Ubuntu script, namely the modularization of autosnort, starting with the web interface choices snortreport and aanval.

I've updated the readme files in each operating system's directory to more of a "release notes" format, that will be easier to read and follow.

One issue that I'd like to write about that I encountered while working on getting Aanval running on CentOS in particular is SELinux. Most people see SELinux as an annoying hindrance that keeps them from getting done what they want to get done. I use to be one of those people as well. But with a little bit of research, most problems with SELinux can be resolved easily, without disabling it or setting it to permissive mode (making it essentially useless).

I ran into a problem during aanval installation where it failed to connect to the mysql database on localhost with the creds I gave it. Thinking I fat-fingered the creds, I did it again and it immediately failed again.

At this point, I check the logs. Nothing interesting in /var/log/messages or the httpd error_logs, so I move to the audit log. For those of you who are not aware, SELinux usually logs to /var/log/audit/audit.log. It's log format is pretty strange, but its easy enough to pick out the relevant pieces to determine if SELinux is interfering with what you you need your system to do. run tail -f on audit.log and try entering the database credentials again and an entry pops up for SELinux, denying access to 3306/tcp for the httpd process. I do a google search for a small portion of the audit log message, and immediately get a relevant result. Long story short, run this command:

setsebool -P httpd_can_network_connect_db 1

 This explicitly states to SELinux that the httpd process requesting connections to the database/database port are perfectly fine. Always remember: disabling SELinux is NEVER the answer.

Happy Easter and Happy Snorting,

DA667

Dropping the hammer on the Aanval (The Aanval Update)

Hello Autosnort Users,

I am proud to announce a new, pilot release of Autosnort that will give users another web interface choice: Aanval!

For those of you who have no idea who or what Aanval is, Aanval can serve as a web UI choice for snort and/or suricata as well as a SIEM for events sent to it via syslog. There's the free edition that supports one sensor, and their commercial releases that support a large number of sensors and/or varying sources.

Aanval is a product of Tactical FLEX -- They're a nice bunch of people. I have yet to meet any of them in person, but I have spoken with both the CEO and the designer of the console over twitter a few times. They are very nice, very personable, and down-to-earth. If you get around to installing Aanval, be sure to drop them a line to let them know how awesome their console is!

This Autosnort release introduces some change you need to be aware of, so let's get to it:

-As indicated above this is a PILOT release. I've tested it extensively over the weekend, but I can't plan for every contingency and problem. Try it out, let me know if you run into problems. The bugs you report help me, other Autosnort users, and the open-source community overall.

-The new version of the script begins an effort for me to attempt to modularize Autosnort a little bit. Individual interface choices are their own child shell scripts now, called from the main shell script. This is a design choice I made to make it easier to add new and troubleshoot existing functionality.

-When I refer to "main" and child shell scripts, I mean, the main script is the large Autosnort script we all know and love. The child shell scripts are much smaller shell scripts dedicated to a single purpose, like installing one particular web interface. For this update, there are two child shell scripts:

snortreport.sh -- as the name implies is responsible for installing Snort Report
aanval.sh -- responsible for installing aanval

-The main autosnort script should be able to be ran anywhere, but the child shell script for the interface you wish to install must be placed into the /root directory for the script to complete successfully.

-As a matter of good practice, I would advise having the main and the child scripts in /root when you run it. The main script expects to find the child script in /root when you make it to the interface choices menu, and call the child script for the interface you wish to install. The script checks to see that the child script completed with an exit state of 0. If it does not, the main script will NOT continue.

-As another matter of good practice, I've decided that from here on out, I will keep the latest version of Autosnort available via github as well as one previous version back. This way, if you find there is some major problem with the current version of Autosnort, and you find that I am not working fast enough to resolve your problem, you can at least try to install the previous version to see if that works for you instead. In the Ubuntu Autosnort directory, there should be a Previous_Rel directory that contains the last version of Autosnort.

I've updated the readme in the Ubuntu directory to reflect this information. Provided there are no major deal-breaking problems in this pilot release, I hope to push support for Aanval to Debian and CentOS in the near future as well.

Update:

Aanval Post-Setup notes:

- It is highly advised that you reboot your system before continuing to the aanval console to continue the installation via the web interface. I ran into a problem prior to rebooting where the aanval console would not recognize that the php mysql module did exist and was loaded until the system was rebooted.

- During the initial setup, aanval will want to know the name of the aanvaldb user and password.

Username:snort
Password:password you gave the snort database user during the autosnort installation

- Aanval has a set of processes that are used to bring events over from the snort database that barnyard2 will dump to, and bring them over to the aanvaldb that aanval reads from. The console interface will let you know if they are not running. To start them, navigate to /var/www/aanval/apps and run idsBackground.pl -start --- I plan on adding an rc.local entry that will do this for you in the near future!

- In order for Aanval to manage events for your snort sensor you need to enable it on the aanval console. click the gear symbol in the lower corner of the web interface. This will bring you to a page called configuration. Click the "Settings" option under the "Snort" section. On the next page, check the enabled checkbox and enter the information for the snort database:

database name: snort
database hostname: localhost
database username: snort
database password: the password you assigned to the snort database user during autosnort installation

then click update. It may take a few minute for intrusion events to show up on the aanval interface. Be patient, they'll start coming in shortly!

- At this time, Autosnort does not support the sensor configuration options that Aanval includes. This is a limitation on my part; I need to figure out where Aanval expects to find the rule files, snort.conf and other configuration files before this will be possible!

- For more guidance and information specific to aanval, pay the folks at Tactical FLEX a visit at aanval.com

Happy St. Patty's day and happy snortin' to ye.