Autosnorby Follow-up: Support to Debian officially released, distributed installations under testing.

Hello again,

As I had mentioned a few blog posts ago updates to autosnort and/or support for snorby were probably going to be staggered out across platforms, since I have to do testing to ensure that things aren't horribly broken on release, that I have the right packages for the right platform that puts the right stuff in the right place, etc.

That being said, porting over the changes to autosnort and support for snorby to Debian went fast. Like, really fast. I should have figured this would be the case since Ubuntu and Debian share similarity in packages installed and package management utilized. It was almost quite literally a copy/paste job -- copy the code from ubuntu autosnort and it its child shell scripts, integrate it in with debian's existing scripts (and some custom things that Debian needs to work right) and just test it to make sure nothing exploded in a fit of errors and incompatibility.

Therefore, it brings me great pleasure to release a new version of autosnort for Debian as well as a snorby child shell script for Debian-based systems, and code improvements to debian autosnort and child shell scripts.

I'm not going to bother going over all of the changes I made to the script(s), since more or less the changes are practically identical to the ubuntu/autosnorby post I made yesterday (e.g. code cleanup, doing things that make sense to make the code easier to manage, installing relevant packages only when they are needed for a particular interface/function, etc.) except to note that I fixed a small bug in the remote database/experimental distributed sensor installation support (In both Ubuntu and Debian)

Speaking of the distributed installation support, I have a rough guide to doing a distributed install. This is very, very rough right now, expect some official or at least "better" guidance soon:

1. Perform a standard autosnort install on your "master console" system, and choose the interface you wish to install. Bear in mind that aanval requires licensing to support more than one snort sensor, and that snort report doesn't really clearly delinate which sensor an event came from. Reboot.

2. Edit /etc/mysql/my.cnf and reconfigure mysql to accept connections globally (e.g. listen on 0.0.0.0; also note that you'd better have a security measure of some sort up to limit connections to your other sensors (e.g. a firewall) if you plan on having this system accessible in *any* way over the internet)

3. run mysql -uroot -p[password here]

GRANT ALL ON snort.* TO snort@'1.2.3.4' IDENTIFIED BY 'my_password'; 

snort@'1.2.3.4' should be the snort database user on the remote sensor you plan to have store events on your master console, likewise the IDENTIFIED BY 'my_password'; should be the password you place into barnyard2.conf to talk to the master console.

4. Run the autosnort installer on your barebones sensor (choose option 2 to NOT install apache or mysql), somewhere down the line, the script will ask if you have a remote database that barnyard2 will be logging into. Enter the username (snort), the remote database name (snort), the remote database ip (e.g. the management interface for your 'master console' sensor), and the password you entered above on the mysql statement on the master console

5. finish the autosnort install (when it gets to the what interface do you want to install phase, just select option 6 for no interface at all.)

6. On the barebones sensor, install mysql-client (apt: apt-get install mysql-client) and try to connect to the database on the master console:

mysql -uroot -psnort -h [master console ip address]
use snort;
show tables;

If you get output, and a list of database tables, that's a very good sign. Otherwise you have some troubleshooting to do. note that by default autosnort installs libmysqlclient to allow barnyard2 to talk to the remote database, however, I'm having you install mysql-client in the event you want to troubleshoot and verify that you can talk to the database as a regular user. libmysqlclient won't let you test things out with manual queries, etc.

I can't re-iterate this enough, but allowing mysql to listen globally opens up some security risks (which is why I've trying to move to having the traffic SSL wrapped and tunneled between sensors/consoles...), so I would most definitely make sure the network your barebone sensor and master consoles are talking over is protected, trusted, and your master console host is firewalled to hell and back, only allowing 3306/tcp connections from your barebone sensors. Know the risks before you take this leap.

Check out the autosnort github and happy snorting!

Cheers,

DA_667

Autosnorby, updates to autosnort, and the updated to-do list

Hello Snort and Autosnort users.

I've got a lot to talk about today, so let's get started.

First and foremost, I've finally managed to work in support for snorby into autosnort. The snorby script installs the necessary packages, a compatible version of ruby (via rvm), all the necessary gems and configures the right files with the right info needed to get snorby up and running. Additionally, I've managed to work out away to drop privileges from the root database user to another user (the snort user in our case)

One thing to be aware of: the delayed_job and the sensor cache jobs do NOT start up on boot. Initial runs of the script had me trying to put commands into rc.local to start the delayed_job and sensorcachejob commands on startup... but they aren't working. I'm working on having a reliable method to start them on boot, but until that is discovered, any time the system is restarted, you'll want to, at a minimum start the delayed_job script:

cd /var/www/snorby && ruby script/delayed_job start

optionally, these commands run the Sensor and Daily cachejobs as well:

cd /var/www/snorby && rails runner 'Snorby::Jobs::SensorCacheJob.new(false).perform; Snorby::Jobs::DailyCacheJob.new(false).perform'

There are options to start the snorby worker process via the web UI, but I find the direct approach to my liking.

Another note is that at first you may notice alerts coming in and displaying properly in the events tab, but the dashboard may not updated at first. If this is the case, on the dashboard page, there is an option to force cache update. Select this option, wait 10 seconds, and reload the page. the graphs should update.

Next up on the list, is the main autosnort script itself. I've done a bit of spring cleaning.

- I've reduced the number of packages installed by default with autosnort.
--By default all we install are the packages necessary to install daq, libdnet, snort, and barnyard 2 with the ability to write to a remote database (more on this in a minute)
--I introduced a quick check that asks the user if they plan on installing a web interface as a part of the autosnort installation, or if they just want a stripped-down barebones install. If you're not sure what option to pick, select option 1.

- I've added a couple of functions to the script near the beginning for pulled pork to facilitate code re-use. this is because:

-- I've added support to download snort rules from previous versions of snort; up to four versions prior.

Why: I ran into a little problem recently where the free rules for version 2.9.4.5 were not available yet, and snort version 2.9.4.6 had just came out. The current version of the script only checks for the current version of snort rules, and one version prior. So now, you have a choice for four versions of rules:

1) the current version (if you have a VRT subscription oink code; e.g. 2.9.4.6)
2) the previous version (the kind that registered users usually get access to; e.g. 2.9.4.5)
3) two more prior versions (in the event that they are needed and/or a similar situation arises)

- If the user has elected to not install a web interface, the script asks them if they have a remote database they want barnyard to log to. I haven't officially tested this feature yet, but theorhetically, this allows for distributed installations... Any volunteers to test this? :)

- The child shell scripts all have functionality to install the packages necessary for them to work individually Again to reduce clutter and excess packages installed by autosnort (e.g. reduce attack surface)

- thanks to DK1844, I have a couple of code cleaniness fixes that I have added, and future todo list items.

I've achieved most of the stuff I intended to do when I initially started this little project, but I still have more that I can do.

Speaking of to-do list items:

- SSL for all web interfaces (generate a default, self-signed SSL cert, use mod-rewrite and mod-ssl to force ssl usage for all web interfaces)
- Further testing with distributed installations (Can I get a master console to register events from a remote system?)
- Encrypted comms between master console and remote sensor (e.g. have barnyard dump database updates to the master console over a secure channel)
- Kali Linux testing! Since Kali is fully compliant with Debian standards, will the debian autosnort script work with it?


The updates to autosnort.sh, snorby, aanval, snortreport and base child shell scripts are being pushed to ubuntu tonight, with support for other operating systems (debian and centOS) to come soon.

Report bugs, problems, praise, feature requests, etc. to:

deusexmachina667 [at] gmail [dot] com

As always the code is on my github repo.

Happy snorting,

DA_667

Another quick post - fix to aanval installer script

Hello AS users,

A recent message from an Autosnort user, sm00th brought a problem to my attention regarding the aanval installer script. I've discovered that there is a problem with the wget command used to download the aanval installer. This is what happens when I attempt to wget the file manually:

root@CG:~# wget https://www.aanval.com/download/pickup -O aanval.tar.gz
--2013-05-13 16:13:59--  https://www.aanval.com/download/pickup
Resolving www.aanval.com (www.aanval.com)... 173.160.180.147
Connecting to www.aanval.com (www.aanval.com)|173.160.180.147|:443... connected.
ERROR: no certificate subject alternative name matches
        requested host name `www.aanval.com'.
To connect to www.aanval.com insecurely, use `--no-check-certificate'.

There is some sort of a certificate problem with aanval.com. The wget command says to try using --no-check-certificate:

root@CG:~# wget https://www.aanval.com/download/pickup -O aanval.tar.gz --no-check-certificate
--2013-05-13 16:14:27--  https://www.aanval.com/download/pickup
Resolving www.aanval.com (www.aanval.com)... 173.160.180.147
Connecting to www.aanval.com (www.aanval.com)|173.160.180.147|:443... connected.
WARNING: no certificate subject alternative name matches
        requested host name `www.aanval.com'.
HTTP request sent, awaiting response... 200 OK
Length: 6703589 (6.4M) [application/octet-stream]
Saving to: `aanval.tar.gz'

100%[=========================================================================================================================================================>] 6,703,589    586K/s   in 12s

2013-05-13 16:14:40 (529 KB/s) - `aanval.tar.gz' saved [6703589/6703589]

root@CG:~# echo $?
0

good news: This allows you to pick up the tarball for installing aanval
bad news: The certificate for aanval.com isn't being checked -- this means we're still using SSL to connect to aanval.com and pick up the package, but we aren't checking the certificate to see who signed it, and/or what site it was signed for.

Unfortnately, the problem is out of my hands. I submitted a quick fix to github.com that adds the --no-check-certificate option to allow us to fix the package.

Regards,

DA_667

p.s. Yes, I'm still working on snorby. Hit a bit of a road bump. I thought I had the full installation down, and now, for some reason the snorby database refuses to update. at all. So, as I said before, I want to run through the entire snorby installation process without a single error. Once I can do that... I'll release the Ubuntu snorby script, then Debian, Then probably CentOS last.

Cheers!

Snorby script progress, and a rundown of how to perform the installation.

Hello Autosnort users,

Still working on the Snorby release for Autosnort, and even then, It's probably going to be a staggered release -- first Ubuntu, then Debian, and Finally moving to CentOS. I imagine that based on some of the issues I have been experiencing, Getting Snorby to play with with SELinux is going to be a lot of fun.

I managed to get it to work with Aanval, snortreport and BASE, so I can do it again.

Anyhow, for those of you who can't wait for me to release a script, here's a basic run-down of the commands I'm running in Ubuntu 12.04 that  I've used that work. this assumes you've ran autosnort up to the point where web interface installation takes place.

Don't be alarmed when it says you're downloading 400m of packages. that's about right:

apt-get install libyaml-dev git-core imagemagick libmagickwand-dev wkhtmltopdf libssl-dev libxslt1-dev libsqlite3-dev libmysql++-dev libcurl4-openssl-dev apache2-prefork-dev default-jre-headless

this installs rvm. rvm is a nice little program that can be used to automatically install ruby:

\curl -\#L https://get.rvm.io | sudo bash -s stable
/usr/local/rvm/bin/rvm autolibs enable
source /etc/profile.d/rvm.sh

this is a cute little hack that I've used for installing the latest stable version of snort, DAQ, and determine what version of snort rules are available via snort.org. From what I gather ruby updates frequently. very, very frequently. This string of commands tells us the latest ruby 1.9.x release available for installation. Why not install ruby 2.0.0? because snorby doesn't support it yet. Submitted an issue to snorby's github. We'll see where that takes us:

wget http://ruby-lang.org/en/index.html -O /tmp/index.html
rubyver=`cat /tmp/index.html | grep -e "ruby-" | head -4 | grep released | grep -v Continue | cut -d"/" -f8 | cut -d" " -f2 | tail -1`
rvm install ruby-$rubyver

After all the packages you need to install, there's also a boatload of gems that need installation. and oh, we haven't pulled down the snorby web UI yet:

gem install thor i18n bundler tzinfo builder memcache-client rack rack-test rack-mount rails rake rubygems-update erubis mail text-format sqlite3 daemon_controller
gem install passenger
update_rubygems
cd /var/www/
git clone http://github.com/Snorby/snorby.git
cd snorby/config

Make copies of database.yml.example and snorby_config.yml.example and rename them to database.yml and snorby_config.yml:

cp database.yml.example database.yml
cp snorby_config.yml.example snorby_config.yml

modify snorby_config.yml. this should be the only change that needs to be made:
sed -i 's/usr\/local\/bin/usr\/bin/' snorby_config.yml

modify database.yml the snorby section at the top, you MUST enter the root database user's password. this is necessary for the rake snorby:setup portion

Next, we have to compile the mod_passenger apache module. this will take a little bit of time.
passenger-install-apache2-module

If the passenger-install command above bombs, try running "gem install passenger --pre" and running the command again.

The passenger-install command will ask you to modify a couple of files. First we have to point apache to the passenger module:

echo "" >> /etc/apache2/apache2.conf
echo "# This stuff is to make Snorby work properly mod_passenger is required for snorby to work." >> /etc/apache2/apache2.conf
echo "" >> /etc/apache2/apache2.conf
echo "LoadModule passenger_module /usr/local/rvm/gems/ruby-$rubyver/gems/passenger-4.0.0.rc6/libout/apache2/mod_passenger.so" >> /etc/apache2/apache2.conf
echo "PassengerRoot /usr/local/rvm/gems/ruby-$rubyver/gems/passenger-4.0.0.rc6" >> /etc/apache2/apache2.conf
echo "PassengerDefaultRuby /usr/local/rvm/wrappers/ruby-$rubyver/ruby" >> /etc/apache2/apache2.conf

Next, we create a virtualhost for snorby and set the documentroot to snorby's public directory, then disable the default site. This effect means that if you navigate to http://[ip address] you automatically get the index page for snorby:

echo "<VirtualHost *:80>" >> /etc/apache2/sites-available/snorby
echo "     ServerName snorby.localhost" >> /etc/apache2/sites-available/snorby
echo "     # !!! Be sure to point DocumentRoot to 'public'!" >> /etc/apache2/sites-available/snorby
echo "     DocumentRoot /var/www/snorby/public" >> /etc/apache2/sites-available/snorby
echo "     <Directory /var/www/snorby/public>" >> /etc/apache2/sites-available/snorby
echo "          # This relaxes Apache security settings." >> /etc/apache2/sites-available/snorby
echo "          AllowOverride all" >> /etc/apache2/sites-available/snorby
echo "          # MultiViews must be turned off." >> /etc/apache2/sites-available/snorby
echo "          Options -MultiViews" >> /etc/apache2/sites-available/snorby
echo "     </Directory>" >> /etc/apache2/sites-available/snorby
echo "</VirtualHost>" >> /etc/apache2/sites-available/snorby





All this has just been setup so we can successfully run bundler and rake to actually install snorby:

cd /var/www/snorby

bundle install --deployment

note: if the bundle install command fails or gives you an error referencing to the gemlock file or a gem called psych shield, you may have to run "bundle install --no-deployment" then run "bundle install --deployment" for things to work right.

finally, the moment you've been waiting for...:

rake snorby:setup

note: do not try to create the snorby database prior to running rake snorby:setup. If the database already exists, rake will bomb and exit.

At this point, provided rake snorby:setup ran, you should have a web interface that you can log into. default creds for snorby are: snorby@snorby.org with a password of snorby.

If you reboot your sensor, the worker jobs and the delayed_run jobs, processes necessary for populating the dashboard on the web UI do not automatically start with the system. You need to run these commands on reboot:

cd /var/www/snorby && ruby script/delayed_job start
cd /var/www/snorby && rails runner 'Snorby::Jobs::SensorCacheJob.new(false).perform; Snorby::Jobs::DailyCacheJob.new(false).perform'

What I am working on:
- Every time I've ran the installation, I've varied one thing (e.g. the first time through I tried installing with ruby 2.0.0, the second time around, I installed and found out daemon_controller was a needed gem that no install guide referenced, the third time around I tried installing by creating the snorby database and granting the snort database user access to the database. This is also when I encountered the problem with bundler complaining about the gemlock file and the psych shield gem and had to re-run bundler with the --no-deployment option.

I need to re-run the installation until I get absolutely zero errors. I want this fully repeatable with no errors.
I want to investigate the possibility of editing the database.yml post-installation: using the snort database user, and using mysql commands to grant the necessary privs to the snort mysql user instead of root. Having the root mysql user's privs in plaintext  in a file that's world readable by default sounds like a terrible idea to me.

possibly adding the delayed_job and SensorCacheJob commands to rc.local to ensure they are ran upon reboot.

Getting support for Snorby is my highest priority right now. After that is done, I have some efficiencies and changes to autosnort I'll talk about in another post. Until then,

Cheers from DA.

Short and Sweet

Autosnort users,

Two things to be aware of:

1) the pulledpork auto rule download functionality appears to be broken right now. This is due to a lack of forethought on my part on believing that there wouldn't be less than 30 days between two versions of snort being released. Working on fixing this as we speak.

2) I'm working pretty hard on automating installation of Snorby. Snorby is an ambitious project and a beautiful web interface, but the Documentation hasn't seen updates in over a year, and things that should be simple now have Ruby, rails and all the dependency hell that comes with it. It's coming together, slowly but surely.

Until next time.

Update: yesterday evening the snort 2.9.4.5 rules were made available to registered users. This sort of invalidates the fix I'm putting into autosnort, but I'm still going to do it to prevent this from happening again. Also still working on a snorby script.

Sometimes the old ways are the best ways (support for BASE and Syslog_Full output)

Hello AS/snort users,

Got two new features in this latest build of autosnort.

First and foremost is support for the BASE web interface. BASE is by and far one of the older and more well-known web interfaces for snort alerting. While the interface isn't going to win for prettiest interface, it is by and far very functional. Some of the cool things include the ability to download alerts as pcap and actual tshark/pcap parsed output for each alert.

This has been added as an installation option in the interface selection menu. The installation is painless and straight forward. After the system reboots, just point your browser to http://[your ip]/base.

The web page initial setup will check for all of the required pre-requisite settings on the first page. On the second page, it will ask you where adodb, a pre-requisite package, is installed. Autosnort and most linux distros install adodb to /usr/share/php/adodb (put that in for the path to adodb). It will also, of course ask you for database information and credentials:
database name: snort
database host: localhost
database port: 3306 (can leave blank if you'd like)
database username: snort
password: (the snort database user's password)

Unless you have an archive database configured, leave that part blank.
On the last page, it will ask if you want to set up auth to get into the web page to review events. That is entirely up to you.

After that, you should be all set to get events to the BASE web interface.

The next feature I have to announce is support for syslog output as another output interface choice. I've added a child script that configures barnyard2 to output to syslog_full format to port 514/udp. I have not FULLY tested this feature, but preliminary tests show that splunk picks up and parses the events with no issues whatsoever. A couple of troubleshooting recommendations:

-Ensure that the sensor's management interface, if it is firewalled, has port 514/udp open and allowed outbound.
-Conversely, ensure that your SIEM or syslog aggregator has port 514/udp open inbound to accept events -- I hosted splunk on a CentOS box and was puzzled why tcpdump says we were sending events out, but they were never making it to be indexed by splunk. I ran system-config-firewall-tui and added a rule for 514/udp inbound (and port 8000/tcp inbound for the actual web UI), and everything was fine.

The child scripts have been made available to Debian, Ubuntu and CentOS.

One last thing:

I've made some minor changes to the script at the recommendation of maintainer of the barnyard2 project. Instead of using sed to modify the copy of barnyard2.conf included with the barnyard2 source, I'm making the script generate a barnyard.conf on-the-fly.

- the script keeps a copy of the original barnyard2.conf in /usr/local/snort/etc/ named barnyard2.conf.origin
- the script generates a barnyard2.conf with only the directives necessary to run barnyard2 (e.g. where are the reference and classification config files, the gen/sid-msg.map files, input format to accept, output format to output to, etc.)
- if you choose to configure output to syslog format, the child script disables output to the database automatically.

As always, the scripts are up on github, along with updated READMEs. Previous versions are available in the Previous_Rel directory as well.

Happy Hunting..

DA_667

New AS release, New init scripts, bug fixes and more!

Hello AS users,

Got a lot to talk about today, so let's get down to business:

First and foremost I've finally gotten around to writing half-decent documentation for autosnort. I've pushed a pdf file to github featuring documentation with screenshots on how to install Autosnort via vmware player, including an interesting hack on how to get bridging to work on multiple interfaces on vmware player 5.

I chose CentOS as the vmware player operating system, but the instructions should be easy to follow for Debian Ubuntu or Backtrack. Any major differences were noted. All said and done it was about 23 pages of documentation with screenshots peppered in.

I focused on using VMware player this time around, and generally speaking the VMware player guidance can be used just as well for VMware Fusion or Workstation. I may do an alternate document for ESXi and/or Virtualbox in the near future.

I've also included general troubleshooting steps and things you *should* see after running autosnort.

Next in line is an update to the autosnort script for Debian, Ubuntu, and CentOS. Banyard2.13 BETA2 has changed how barnyard2 behaves. As a result of this, I had to change how arguments are passed to barnyard2 in Autosnort. The details are available for each operating system's readme files. Suffice to say, everything works fine now.

Finally, I've gotten around to testing and posting init scripts for Ubuntu and Debian to control snort and barnyard2. The Ubuntu and Debian init scripts support start, stop and restart functions and are pretty easy to implement. Implemtation details are also available via the Readme files for Ubuntu and Debian Autosnort scripts, but suffice to say, its incredibly easy.

I may release a modification to the init script that detects if you installed aanval and automatically start/stop the BPUs along with snort and barnyard2, but that will come a little later.

The documentation, new scripts, readmes and init scripts should all be available via github as of now.

Cheers,

DA