Installing Suricata 2.0 on Debian with JSON support

A few months ago, I wrote a blog entry worth reading if you lack context on installing Suricata on Debian with banyard and syslog support. During my original research, Suricata 2.0 was under development but has since been released. What is very interesting about this is its JSON support.

I am using Debian Wheezy but I did not want to install from sources nor upgrade to sid. So I ended up recompiling libhtp1 and suricata 2.0 for wheezy, which you can now download right here:

These packages have been compiled on my own wheezy server with latest patches installed. I have also installed libjansson4 and compiled suricata 2.0 with JSON support (Debian suricata on wheezy disables JSON support by default).

It will then allow you to output JSON directly to a file, I advise you to take a look at that link on Logstash Kibana and Suricata JSON output if you need more information on how to enable this with your logstash/kibana installation.

You can install some nice dashboards for suricata to use in Kibana, I have been using these ones lately.

I will write a bit later on tidying your alerts.

A week with the Tado

Well almost, in 2 days it will be, but I feel I have enough evidence to write up a quick review of the beast. So Tado is a concurrent to nest which you can purchase for most houses/apartments depending on the boiler you have. It is a thermostat unit for your boiler. It is highly recommended you go through the site to confirm that your setup is compatible.

My device took a while to arrive because after Christmas, Tado became really popular and they started to have an impressive (several weeks) backlog of orders. Eventually I received this.



I had a little trouble with the wiring due to the fact that my house is not exactly what you would call standard, so when I followed the wiring examples from the website, it just did not work for me. This is also because I can only heat water at the same time as the boiler is on, there is no separated pipes. After dealing with support, they were a bit slow in providing me with the right wiring, I ended up giving a call to my electrician to figure this out, the result being here.


Once this is wired right, you can put the cover back on and check on the website that all the bits are working. In fairness, apart from the minor setback in wiring, it is a piece of cake to install. You register the device with the user/password written on the card in the box.
You can then setup our phone or any other tablet to register with tado, this will allow tado to use geo location to save on power/heating when you are outside your home. The website interface reports at all times on the current temperature inside and also the weather report that it is supposed to use for intelligent heating.


The reporting is fairly straight forward, it is able to tell you at a glance how much heating it had to do (dark blue patterns on the graph), how long you were away on a given day and other useful information. For heating, there are two main settings for the device: savings or comfort. Savings is when you want to spend at least as possible on your bill, so it will not heat straight away when it detects you are moving home whereas comfort will be much snappier to react. I chose savings because I do not mind that the house will take slightly longer to heat up.


You can also set temperatures for normal operations and sleep, on the web interface or your phone. The phone application is quite responsive and intuitive, it will notify you of current status and will change colors to let you know which mode is it in, like below in away mode when you step out of your place.


It will also change context when the night comes, depending on when you have set up your sleeping hours. I have decided to go for custom times depending on the days as I do not wake up at the same time regularly. You can also override all settings to manual and just heat up the place if you need to, there is a button on the device itself, or you can control this using the application or the web interface.


I bought the tado to finally replace my timer with a clever thermostat and it does just that and very well. In the end, I am unsure if it will cost me a bit more than it used to mostly because i used to heat up the house 3h per day, and I think at the moment it heats up for slightly longer than that. That said, having a house that is at the right temperature all the time and reporting on energy consumption is quite good.

Very happy about the purchase. I have asked about a timer functionality for summer so i can heat up my water a couple hours a day but no response so far.

Installing Suricata, Snorby and Banyard2 on Debian

I have used Snort quite extensively in the past and was curious about toying with Suricata which is similar to Snort but nicer in my view. It has been a few years since I looked at it. I can see the project seems to have evolved quite a lot. One functionality that I will be using down the line will be PF Ring.

On a lazy Sunday afternoon, I thought this was the perfect time to take a look at what it can do in its current form. I used Debian 7.3 for my tests. Everything is packaged which is quite nice though the version of suricata is a bit old on this (1.2.1 vs 1.4.7 on the website). I am very likely to make packages for this later in order to have more functionality.

Once you have done the traditional apt-get install suricata, there is not much to do to get it running, mostly edit: /etc/default/suricata and change this line depending on your network interface, and also allow it to run:

# set to yes to start the server in the init.d script
RUN=yes
# Interface to listen on (for pcap mode)
IFACE=br0

You then should grab the rules to get it all going and monitoring, check out the official page to set this up. I edited /etc/oinkmaster.conf to add the rules I wanted:

url = http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz

You now need to grab the rules, a quick mkdir /etc/suricata/rules && oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules should fix this, and give you something like this:

~ # oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules
Loading /etc/oinkmaster.conf
Downloading file from http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz... done.
Archive successfully downloaded, unpacking... done.
Setting up rules structures... done.
Processing downloaded rules... disablesid 0, enablesid 0, modifysid 0, localsid 0, total rules 18195
Setting up rules structures... done.
Comparing new files to the old ones... done.
Updating local rules files... done.
[***] Results from Oinkmaster started 20140119 18:15:26 [***]
[*] Rules modifications: [*]
    None.
[*] Non-rule line modifications: [*]
    None.
[+] Added files (consider updating your snort.conf to include them if needed): [+]

    -> botcc.rules
...snip...
    -> unicode.map

Restart the thing with a simple service suricata restart and there you are, you can leave it running on your system to learn what kind of traffic is happening. It is worth noting that default rules are set to PASS to avoid messing your traffic up. It is up to you to tune this the right way(tm).

Snorby is a web interface that allows you see events in a nice web inteface. It will require a few things to work nicely, which you can install prior by doing: apt-get install bundler libxml2-dev libxslt-dev libmysqlclient-dev graphviz-dev libgv-ruby wkhtmltopdf.

Before you execute the next commands, be careful with your snorby_config.yml file and set your domain to a secure domain and random port, since this is a ruby on rails application, unless you plan on proxying it behind a http server. My 2 cents, opinions my own, etc…

cd /opt
git clone http://github.com/Snorby/snorby.git
cd snorby
bundle install
cp database.yml.example database.yml
cp snorby_config.yml.example snorby_config.yml
vi snorby_config.yml
cd initializers/
vi mail_config.rb
bundle exec rake snorby:setup
bundle exec rails server -e production

Now you need to set up a parser between the suricata logs and the snorby interface, this is where banyard2 comes in. The new version is hosted on github. You will need a few things to get it compiled right.

cd /opt
git clone https://github.com/firnsy/barnyard2.git
cd /opt/barnyard2/
apt-get install dh-autoreconf libpcap-dev
# check out where your MySQL libs are before specifying the same folder
./configure --with-mysql-libraries=/usr/lib/x86_64-linux-gnu/
make 
make install

If there were no errors, you should have a nice running setup, time to configure it to send stuff to MySQL. Edit /usr/local/etc/barnyard2.conf and change the following:

# set the appropriate paths to the file(s) your Snort process is using.
config reference_file:      /etc/suricata/reference.config
config classification_file: /etc/suricata/classification.config
config gen_file:            /etc/snort/gen-msg.map
config sid_file:            /etc/snort/community-sid-msg.map

# define the full waldo filepath.
config waldo_file: /var/log/suricata/suricata.waldo

# database: log to a variety of databases
output database: log, mysql, user=snorbydbuser password=snorbydbpassword dbname=snorbydbname host=localhost

You should then be able to start it and check that it works, if it does, then you can use -D to run as a daemon.

touch /var/log/suricata/suricata.waldo
 1234  barnyard2 -c /etc/suricata/barnyard2.conf -d /var/log/suricata/ -f unified2.alert -w /var/log/suricata/suricata.waldo -d

More on this when I have time :)

Installing emoncms on your home server

I have started to look into home automation and graphing energy consumption. There is a plethora of proprietary solutions out there which of course are in my view not ideal. I stumbled across this project: OpenEnergyMonitor which seems to tick all the boxes. The main criteria for me is OpenSource so I can export my data any way I see fit.

I have placed an order on some hardware that they sell, initially to monitor my electrical consumption but also temperature and humidity levels. I am planning to add central heating controls at a later stage.

This first post is to breakdown the installation of the software on a normal Linux server. Obviously, I am assuming you know Linux a bit and you already have a LAMP installation somewhere. The following instructions are for CentOS 6.x, you can also find full instructions here. Also, I am not touching on security, so it is up to you to secure your setup any way you see fit.

You will need redis and MySQL especially based on this test.

You can start by installing the required backend and checkout the latest sources:

git clone https://github.com/emoncms/emoncms.git

Then copy the default settings file in the folder and edit to match your MySQL configuration (you will need to create a default database for the project in MySQL). Then change the following settings:

  $username = "yourluser";
  $password = "yourpass";
  $server   = "yourserver";
  $database = "yourdatabasename";
  $default_engine = Engine::MYSQL;

You will need the EPEL repository to install redis, like doing so:

yum install redis php-redis
service redis start
chkconfig redis on

This should allow you to head to your vhost on your local server and register for a new account, this will by default become an admin account, so treat accordingly.

I will add more on redis and the setup of the devices whence I receive them.

Nexus 5

It was only a few months ago (February 2013) when I got the Nexus 4, a device I truly loved. Being the Android fanboy that I am, when Google uncovered the Nexus 5 at the beginning of the month, I could not resist. In fairness, if you compare them side by side, they might not be that different after all, but what mostly caught my attention was a promise for much faster CPU, better and bigger screen with 1080p resolution, better camera. That is about it.

I read two reviews before deciding to order, the one from EnGadget and the one from The Verge. My main gripes with the Nexus 4 were the camera and the screen quality. I am glad to report that these have been fixed, at least to my standards.

Again, you get google stock Android which is nice. Kitkat needs a bit of tuning though which I expect to see happening quickly. Once you tame the Location service and disable Google Now, the device performs wonderfully on saving battery. In my normal day to day usage, I reach something similar to the Nexus 4, about 2 days uptime and 5 hours screen on time.

Google also managed to create a very sexy device. The Nexus 5 feels nicer in hand than the previous Nexus, managing to go beyond my expectations on this one. The device is also amazingly fast, like really blazing fast. This is quite amazing, the change in CPU really makes the difference. I will be curious to see how fast the Nexus 4 will be on kitkat whence the update is available.

So, no regret.

Fix you laptop GPU with a 26 euros heat gun

This post exists thanks to a friend of mine, Alexandre. I have an old trusty Sony Vaio laptop (FZ11Z) which one day froze while reading a video then displayed green and purple lines. After a few reboots, that was permanent, happening from bios screen to Windows. A good way to find out what was going on is to go into Device Manager where it reports an error 43, which means that it disabled the device acceleration and reverted back to a plain VGA driver. This is how it looked like when faulty:


I was about to order a 200 euros motherboard to replace this since my diagnostic was busted GPU which is unfortunately soldered onto the motherboard. This is when Alex made me aware of a condition that affects most Sony laptops of that generation.

The fix is to get a heat gun, I got this one in Argos. You will need to take the cover and screws out to expose the back of the laptop. You will need to unscrew all the heat sinks around the CPU and the GPU to expose them. The NVIDIA GPU is the one on the right handside:


Make sure you cover everything around where you are going to blow hot air to avoid melting any other parts. I used 2 layers of cling film (alloy), like this:


Heat this with a heat gun for about 2mn at 300 degrees celsius. Point the gun downwards with a distance of at least 15cm, to avoid burning the chip, then let it cool down for at least 30mn.

This is it, fixed my laptop in no time, here is a video of a boot after the manipulation :) Fixing a motherboard with a 26 euros tool versus ordering a new motherboard for 200 euros, feels good :)

Safely overclocking your Raspberry Pi

Safely might be a slight overstatement but I have been in the overclocking business for a long time now. I did not really consider overclocking my Raspberry until I read some post than claimed that OpenElec (which is what I use) had gained some serious speed by doing so.

My hardware is a Raspberry Pi Type B, so your mileage may vary, you should always be extra careful before changing stock settings. You should also read the overclocking guide. Unlike the copy/paste values in the link, I have not over volted the device because, it is very bad for hardware if you don’t need to and, you will lose your warranty.

So I ended up picking slightly safer settings, you will first need to SSH to the device then remount the flash partition with write permissions

mount -o remount,rw /flash

Then edit the file in /flash/config.txt and uncomment the following lines and change the values like this:

arm_freq=850
core_freq=325
sdram_freq=425

Then save and reboot the device. You can check the running speed of the CPU by doing cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq or vcgencmd measure_clock arm.

The beauty of it is that it is dynamic overclocking so only when the device needs speed will it boost, especially during boot, scan of libraries and parsing of medias. When playing movies, unless it lags, it will remain at 700Mhz.

Happy watching!

gmvault, claim your mail back.

It has been a while since I have written a post on this blog. I recently got back into looking at how to backup my gmail accounts, just in case, because paranoia, you know…

As I thought I had seen back then, you cannot use Google takeout. I am unsure why exactly but the end result is that I needed a solution. There are a lot of defunct projects out there but an active one that is quite cool: GMVault.

Not only can this thing backup your mail on regular basis (note that if you use 2 factor auth, you will need to create an application password to allow gmvault to download all your mails), it can also export them to mbox/maildir.

You can of course take a look at the install guide for further details, i just went for the source install (using FreeBSD here) by doing:

tar xvf  gmvault-1.8.1-beta
python setup.py install

You can then start using the program straight away, to back your mail account up, just use:

gmvault sync --passwd emailaccount@gmail.com

Once this is finished, you can easily export this onto a Maildir account:

gmvault export -d location_of_your_gmvault-db -t maildir /where/to/export/

This worked well for me bar one issue, my dovecot/roundcube install decided not to see the folder, this fixed it:

for i in `ls -d */`; do mv $i .$i; done

I have donated money to this project because they are worth it, if gmail shutdowns for me tomorrow, i have a full backup of my mails on a server I own.

New shiny server!

So imagine my surprise when a friend gave me a link to my current hoster and got me to find a nice server with twice the amount of RAM, twice the hard disk space, bigger CPUs for 5 euros less per month :)

And this is how you are now reading this on the new server! I took the opportunity to migrate from Debian 7.0 to FreeBSD 9.1 since I wanted to go back to FreeBSD for a while.

Loving the speed and glad to BSD a bit more! Hoping I can post more than I did over the last few months.

FreeBSD behemoth 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #0: Mon Jun 17 11:42:37 UTC 2013

Booting XBMC OpenELEC with NFS storage

I have had a raspberry pi type B for a few weeks now and one problem that I have is on the SD card which can get stressed and corrupt XBMC data. I was looking for a solution tonight and eventually stepped on it reading a few pages from the wiki. Since I still want to use the SD card, I just changed /flash/cmdline.cfg to contain this:

ip=dhcp boot=/dev/mmcblk0p1 disk=NFS=192.168.0.66:/media/xbmc

Of course, you will need to change the NFS server and mount point as they are likely to be different. Worth noting that I rsync’d the content of /dev/mmcblk0p2 (/storage in OpenELEC) so I would not have to scan my whole collection again.

There is also a full procedure to netboot your Raspberry Pi but this did not appeal to me, details there.