Monday, August 26, 2013

Project H1N1: An introduction.

Hello Autosnort viewers, today I'm going for a change of pace and going to introduce a bold, new project that could have a monstrous impact on the IDS and Network Intelligence realm. I'd like to introduce groundbreaking for the new Unallocated Space collaboration project, H1N1:
----------
H1N1 – Abstract:

H1N1 is a project that is meant to integrate the PRADS Passive Operating System Detection with Snort, the popular Intrusion Detection System package. The end goal is a system where Passive OS fingerprinting and IDS both work off one another to present a bigger picture:

-    Snort benefits would benefit from Passive OS fingerprinting by having the Passive OS fingerprinting software inform it as to what operating systems are in its HOME_NET, and use that information to build Preprocessor policies, specifically frag3 and stream5 reassembly policies, that control how snort handles reassembling traffic for those hosts. Eventually this gathered information could be used to make snort rule recommendations based on hosts, client service banners observed in the HOME_NET as well.

-    PRADS would (eventually…) benefit from snort, by existing as a preprocessor in the “stack”, benefitting from snort’s native ability to reassemble fragmented packets and reassemble TCP streams for both Operating System identification and Client/Server banner grabbing from the wire.

-    Users would be able to review information for assets in their HOME_NET detected off the wire, alone with their intrusion events, offering greater network intelligence and a bigger picture.

The problem is, I’m a complete derp when it comes to anything programming or development. I know shell scripting, I’ve done some introductory python, but beyond that, I’m pretty much helpless. The key to completing this project lies in the Unallocated space motto: Teach, Learn, Build -- We'll teach one another something, We will learn in order to fill in the gaps of knowledge we may lack in making this project a reality, and the end goal will be to Build something awesome.

So that's the general abstract and overall goal of this project. Yesterday afternoon, I did a presentation about the project at my local hackerspace, Unallocated Space  The goal was to find others in my area who would want to collaborate and be interested in making the project a reality. Below is an overview of the discussion and the powerpoint slides I brought with me:
 ----------
H1N1 – Day 1: Introductions, Teach, Learn, Build

The first day was primarily an introduction to the technology behind the project. Unallocated motto was integrated into today’s talk, in introducing H1N1:

-    Teach:
o    I introduced the audience the open-source IDS/IPS software snort, if they weren’t already familiar, as well as how it worked. After that I explained the difference between Active and Passive OS fingerprinting, then delved deeper into the various techniques used to passively fingerprint operating systems that have been researched and well-established by minds much greater than my own; such as TCP SYN flag options, Client/Server banners, ICMP fingerprints, etc. I discussed the weaknesses of Passive OS fingerprinting – network devices modifying traffic between the source and destination, duplicate fragments/segments, etc.

o     I then set the stage for H1N1 – why hasn’t there been a project that integrates these two technologies together and have them work together? Most of the more well known solutions are commercial, while some of the Open-Source solutions aren’t quite there yet, in terms of usability .

-    Learn:
o    I want to make this project a reality. But I next to 0 programming knowledge. I know how to “hello world” in three separate languages, but unless I have an active challenge, or a reason to extend my knowledge beyond that, I tend to lose interest and my knowledge drops off nearly immediately. H1N1 is going to be a challenge, A challenge that I don’t know where to even begin to tackle.

o    This is where Learning comes into play. I want to learn how to develop in different programming languages. I’m more than willing to learn from anyone willing to teach. If you know how to do it, show me. If you don’t have the time, tell me to “Go learn about X concept in Y language then come back to me.” Once I understand it, I’ll teach others. We all benefit and gain knowledge in this way. Knowledge is power, and we all deserve to wield it.

-    Build: The project is ambitious, but I want to start with baby steps. Let’s start with the basics.
o     What passive OS identification software do we want to use?
o    Where are we going to host the code, and eventually the alpha builds?
o    Let’s make sure that Snort and PRADS can actually sniff off the same interface without causing the box to explode?
o    The current Prads2db script is in perl… Hell, most of the prads utils are in perl. Should we fix that? Can we make them better?
o    We can output the prads data to a database, to host attribute XML, but currently, there’s no web interface to display this data on.
o    (This is probably something that will happen much later) Is there host timeout functionality integrated in with PRADS? If not, how can we make it?
o    Automating the prads2snort script (rewritten in something that isn’t perl), and having it run on a regular interval, given to snort, and passing snort SIGHUP to read the new hosts identified in HOME_NET
----------
Finally, We ended the conversation with a review of the slides, Q + A and what direction to take the project -- a review of the Build portion of my slides where I wanted to establish project milestones for H1N1:
----------
H1N1 Q + A:

o     What passive OS identification software do we want to use?

Answer: We all primarily agreed that PRADS has the most features out of the options presented, and it made the most sense to build off it.

o    Where are we going to host the code, and eventually the alpha builds?

Answer: For the time being, we all have infrastructure in our own homes that we can build snort and prads on, Forgotten offered to create an unallocated space github account for hosting the project code there. Eventually, when we have a workable alpha release, or beta release, or SOMETHING, we can talk to hunter and UAS Keyholders to either get a physical box or virtual machine to demo things.

o    Let’s make sure that Snort and PRADS can actually sniff off the same interface without causing the box to explode?

Answer: This should be very easy for us to do. compile/build a snort sensor (snort, barnyard2, a web interface of choice), then compile PRADS. Set them both to run on startup (either via rc init script, or via rc.local), throw traffic at it. If it doesn't explode in a fit of rage, I'm going to call it a victory.

o    The current Prads2db script is in perl… Hell, most of the prads utils are in perl. Should we fix that? Can we make them better?

Answer: There's a general fear and loathing of perl that's understandable, along with a love of python. Additionally there's been talk at our first session that it would be trivial to write a parser to drop this information into a database. Any volunteers for reviewing the prads2db.pl script as well as the prads2snort.pl script, and prads.sql data and determining whether or not porting it all to python would be feasible?

o    We can output the prads data to a database, to host attribute XML, but currently, there’s no web interface to display this data on.

Answer: Ruby on rails has been suggested. I don't know enough about web application technology to make this happen, but I'm not opposed to trying this. Another technology that I've been curious about is node.js.. A lot of new up and coming infosec projects have used node.js, and it's supposed to be incredibly fast. I wanna learn more.

o    (This is probably something that will happen much later) Is there host timeout functionality integrated in with PRADS? If not, how can we make it?

Answer: It was suggested that we turn this into a rolling time window sort of thing that's user configurable. Instead of taking the first fingerprint we see and associating it to an IP address, we can keep fingerprints associated to an IP address seen over a period of time, and based on results over X time period, Have Y% confidentiality that this IP address is this type of operating system. This is likely something that will have to come much further down the line, as it seems to be something that will require considerable effort to code.

o    Automating the prads2snort script (rewritten in something that isn’t perl), and having it run on a regular interval, given to snort, and passing snort SIGHUP to read the new hosts identified in HOME_NET

Answer: This should be fairly straightforward to do: Basically read the database, and generate a host_attributes.xml the same way the current perl scripts read the prads log and generate output.

----------
That just about sums up the discussion about this project. This project is still very much in its infancy. No hard timelines for ANY of the milestones established have been set. The reason for that is that I want this to be a project that anyone can come into with any level of programming knowledge, say "this part of the project interests me", take charge of it, learn the concepts necessary to complete it, come back to the group, teach what they learned, and implement in a way that helps meet the overarching goal of the project -- Teach, Learn, Build. To make this happen, setting hard timelines isn't feasible since everyone learns at different rates.



I hope to drum up support for this project as far and as widely as I possibly can. I'm not looking for anyone to do the code and solve the problems for me, or folks wanted to collaborate on this project. Even if your contribution to this project is "This task would be best suited to this programming language. Go learn about this concept in this language and come back to me."

A good friend of mine, Forgotten has started a collaboration github account on github, by the namesake of this project:

https://github.com/Unallocated/H1N1

Currently there isn't much there other than the readme; I'm hoping to change that if not on my own, then with your help, slowly but surely.

Hope to see you on the list of contributors,

DA_667

Friday, August 23, 2013

autosnort: compatible with Linux Mint

This is a really quick post. I've installed Linux mint on my person desktop at home, and out of curiousity, decided to try to run autosnort on mint to verify whether or not any of my build scripts were compatible. Well, the ubuntu autosnort script is 100% compatible with mint.

I'm aware that mint is primarily a desktop operating system, but if you're interested in running it on a Mint desktop as a sort of personal single workstation IDS, here's what to do:

1. Get a root shell: sudo /bin/bash
2. wget http://www.github.com/da667/Autosnort/Archive/master.zip
3. unzip master.zip
4. copy the autosnort-ubuntu-[date].sh script and an interface installation script to /root
5. run "bash autosnort-ubuntu-[date].sh"
6. run through the prompts as normal.
7. At the very end, when the script asks you whether or not you want to reboot, choose NO.
8. sudo vi or sudo gedit /etc/rc.local and modify the ifconfig eth0 line.
It initially reads: ifconfig eth0 up -arp -multicast promisc
make it read: ifconfig eth0 up promisc
9. save the file.
10. either reboot, or run "sudo bash /etc/rc.local"
11. Open firefox, or the web browser of your choice. type "localhost" in the address bar. You should be greeted by the web UI of your choosing.
12. ps -ef | grep snort will confirm whether or not snort and/or barnyard2 are running.

Yes, I know I've said in the past that managing your IDS on the same interface you're sniffing on, or having your sniffing interface be addressable isn't a good thing, but those are for DEDICATED IDS installations, this little "hack" is meant for users who want to drop snort on their personal workstation as an added layer of security.

Cheers,

DA_667

Sunday, August 18, 2013

Changing of the seasons; RHEL/CentOS code push/update; Autosnort milestones posted.

    So here we are, at the end of August. Defcon has came and went, summer vacation is coming to close, and labor day is right around the corner. I don't consider summer officially over until Labor Day has come and passed, but why postpone it?

On that note, I've been busy in the lab, and have a whole host of updates that I'm dropping tonight for Autosnort. 99% of these updates are specific to the RHEL/CentOS release, that has been long neglected -- much longer neglected than the other distro scripts. The last update you all saw for RHEL/CentOS was sometime in April. The last major blog post you saw was in regards to the 9 layers of hell I traversed to make Snorby work on CentOS with SELinux enabled, then I disappeared for summer break. So, without further adieu, here's what's happening:

RHEL/CentOS:
- Updated the entire look and feel of the main autosnort installation script. CentOS/RHEL users now have the metasploit like prompts just like the Debian and Ubuntu users. Only things the user should be aware of are printed to the screen now:

    - Status updates are in blue (e.g. [*] this indicates what autosnort is doing currently)
    - Notifications are in yellow (e.g. [*] this indicates something the user needs to pay attention to (such as a prompt, or something they may want to note somehow))
    - Successful modifications are in green (e.g. [*] this indicates whatever autosnort was doing was successful)
    - Unsuccessful modifcations/installations are in red (e.g. [*] something bad happened. You'll probably want to get a hold of me and give me the log files for your installation, so it can be resolved)

- Speaking of logging, the installations scripts no longer spew output all over your screen buffer. Thanks to neat trick I picked up from stack exchange, the output of every major command is saved in two separate log files in /var/log:

    -/var/log/autosnort_install.log -- contains output from all the major commands ran from the main autosnort installation script
    -/var/log/[interfacename]_install.log -- contains output from all the major commands ran from the interface installation script for the interface you chose to install.

- The main autosnort installation script and all the web interface installation scripts have been updated with the new metasploit-like prompts and the background logging. This includes:

    -autosnort-CentOS-[date].sh
    -snortreport-CentOS.sh
    -base-CentOS.sh
    -aanval-CentOS.sh
    -syslog_full-CentOS.sh
    -snorby-CentOS.sh

- All web interface installation scripts for RHEL derivatives have had their DocumentRoot and Directory directives reconfigured to serve out the web interface of your choice. This means all you have to do is point your web browser to the ip address of your sensor's management interface, and provided you reconfigured ip tables to allow port 80 inbound, your web interface will pop up automatically.
- All web interface installation scripts for RHEL-based distros are 100% compatible with SELinux
- All web interface installation scripts for RHEL-based distros have had the ownership of DocumentRoot changed to the apache user and group
-Fixed minor grammatical and syntactical errors littered throughout the script.    

Ubuntu/Debian:
- Apparently at some point between now and june, the passenger output directory for the mod_passenger.so binary changed the name of the directory from "libout" to "buildout". sigh. consistency is awesome, don't you agree? I only discovered this during testing passenger during the CentOS testing process.
- In an effort to make the mysql installs uniform between all autosnort builds and promote better security, I've made the mysql-server installation for Ubuntu and Debian silent, but now, just like with the CentOS script, the /usr/bin/mysql_secure_installation script is ran as a part of autosnort. huzzah for better secured databases.
- Same as the CentOS script, found minor grammatical and syntactical errors littered all over the script. Found and fixed what I could.

Code push should be happening tonight before init 1 for the night. As usual, the scripts are open source and are available on the autosnort github:

https://github.com/da667/Autosnort

While we're on the topic of github, I've posted some milestones that officially lay out where I want to take autosnort next -- things I want to accomplish with it. Have a look, see if I'm missing something:

https://github.com/da667/Autosnort/issues/milestones?with_issues=no

And last, but not least... The new project ideas I spoke of in my last blog post? I'm holding a meeting at our local hackerspace about one of them in particular, the passive OS fingerprinting project, trying to get some helping hands with it and make it more of a success. If you're in the Maryland/DC/VA area, come to unallocated space on the 25th so we can get a project outline hashed out.

Cheers,

DA_667

Monday, August 5, 2013

End of my Summer vacation; new projects to consider -- want feedback!

Dear autosnort users,

I just got back from Defcon. Summer is almost over. You're probably wondering if this lazy louse is going to get off his ass and start scripted again. Well, yes, but, I have some details to announce.

1. I will definitely be continuing the Autosnort project. I never really had the intention of taking the summer off, but now I'm kinda realizing that it makes sense to do this. Summer is practically the only time I get to spend with other hackers locally and at Defcon, and one of the few times a year that I get to see family back home.

First on my order of things to do is to finish testing on CentOS for pulled pork. I released the selinux module I painstaking build through a couple of weekings of gnashing teeth and cursing, but I want to have the selinux module generated by the snorby install script for centOS instead of having use have to bring another item with them for the installation. Right now, I have the script echoing out the source of the module that I got to work and building that generated file. It looks kludgy in the code, but it works wonderfully.

2. I have an ambitious project that I really want to get some help doing. I was a former Sourcefire employee. That being said, I'm familiar with RNA or what they call firesight right now. The general gist of this technology is to gain insight as to what operating systems are running in your network, and make one's snort ruleset more efficient based on that -- You're not going to care about X11 exploit rules if you're not running Linux boxes with publically listening X servers, the same as you wouldn't care about seeing alerts about SMB/windows malware attacks if you have your HOME_NET defined as a network that is solely a groupe of freeBSD servers not running samba.

p0f -- passive OS fingerprinting has been around for years, at least a few decades. I was present at shmoocon. The project had been idling for a while up until a group of folks decided to request access to oversee the project and continue on with it. This was a talk at shmoocon over a year ago. The project is alive and kicking.

What I want to be able to do: take p0f signatures, have them monitor a defined network, and develop a measure of confidence as to what is running in your home network.

take this intelligence and use it modify pulledpork -- enable/disable sids based on p0f results, set ip defragmentation and tcp stream reassembly policy by majority of X hosts in your HOME_NET, etc.

the thing is, I'm not quite sure where to start. If anyone wants to help me with this in any way... I want to make this happen.

3. Malware analysis wiki. this inspired by my friend @forgottensec. Forgotten ended up providing the Capture the Flag community the CTF wiki. Well, I want to provide a resource for the information security community as well:

There seems to be information scattered to the seven corners of the  internetz. I want to gather as much information as I possibly can into one way for malware analysis techniques. For example, I don't the first thing about using debuggers to debug malware, but I know lots of interesting things to look for, when it comes to dynamic malware analysis. Why not put it some place publically accessible and easily contributed to?

Let me know what your thoughts are on these new developments I'm thinking about, until then... I'm gonna enjoy the end of my summer before I have to get my lazy ass back to work