Many organizations out there, big and small are running SIEMs (Security Information and Event Management) solutions. There are many providers out there: HP ArcSight, IBM Q1 Radar, Splunk, McAfee Nitro and so on. I think they are invaluable in an organization trying to monitor what is going on at any given time on their network. For those of you who do not know what a SIEM is, well think of it as a bucket where all you log information trickles into. SIEM technology provides real-time analysis of security alerts generated by network hardware and applications. SIEM solutions come as software, appliances or managed services, and are also used to log security data and generate reports for compliance purpose. A SIEM provides capabilities such as:
- Data Aggregation: Assist in aggregating data from many sources, including network, security, servers, databases, applications, providing the ability to consolidate monitored data to help avoid missing crucial events.
- Correlation:looks for common attributes, and links events together into meaningful bundles. This technology provides the ability to perform a variety of correlation techniques to integrate different sources, in order to turn data into useful information.
- Alerting:automated analysis of correlated events and production of alerts, to notify recipients of immediate issues. Alerting can be to a dashboard, or sent via third party channels such as email
- Dashboards: tools take event data and turn it into informational charts to assist in seeing patterns, or identifying activity that is not forming a standard pattern
Now, lets take ArcSight for example (as this is the one I am most familiar with). ArcSight gets all these logs from disparate systems and converts them to a common format called CEF. or Common Event Format. This format contains the most relevant event information, making it easy for event consumers to parse and use them. This CEF data is then used by ArcSight to do its magic, which I won’t get into in this post. Now, a SIEM is a great tool to add to your SOC or IT security team’s arsenal, as it provides visibility over what is going on (depending on how many devices you monitor). However, what I have noticed, is without some form of intelligence fed into your SIEM, you will be chasing your tail endlessly. Specifically, how do I know what is good and what is bad. Now, there is a lot you can do with a SIEM operating in a “bubble”. Looking for things like people from not so trusted countries (use your imagination) trying to penetrate my external perimeter over ports like TCP/1433, TCP/22, or TCP/3389. DNS traffic from your internal network to DNS servers in again, those not so trusted countries, alerts from your enterprise A/V systems, etc. These are a good start, you will notice eventually that you wish your SIEM could help you more.
Imagine the situation where your A/V solution is not detecting certain types of malware (i.e. FakeAV) – say it isn’t so! How could you potentially use your SIEM to detect malware? Well there is a way. It is not perfect, but it really does help. The key is utilizing intelligence sources to “supercharge” your SIEM. Taking the intelligence and having your SIEM monitor for hits. For example, if you are pulling your web proxy logs into your SIEM you can match your traffic against sites like Malware Domains List. When you get a match, you know that a user if potentially visiting site where malware is being hosted. The key is a combination of the intelligence and what visibility you have over your network.
These sources can from two places. 1. Manual, and 2. Automated.
- Manual – These are simple. As you identify various information sources, you plug them into your SIEM. The real manual part of this is not the “plugging” part, but the reading and hunting for the intelligence. So reading blogs like and looking at sites like Malware Domains List. This is tedious as you are constantly having to follow what is going on. This isn’t bad, but time-consuming.
- Automated – This is the preferable method. Setting your SIEM up to use a stream of intelligence and monitoring your logs for any matches. So essentially, using ArcSight as the example, all you would need is a feed that will produce native CEF or be able to shoot your data into a syslog feed to be picked up by ArcSight. In many cases there are solutions that will do all the work for you. Let’s look at a few:
- ArcSight Open Source Intelligence Utility (ArcOSI), now called “Bad Harvest”. The free code is still available – but it has gone commercial, so I don’t know how long this will be available. The new company, Threat Stream are now offering tools such as Honey Cloud, Sink Cloud, and Open Source Intel. So ArcOSI or the new Open Source Intel is fed by a collection of 70 sources of open source intelligence, with new sources are added regularly. It pulls lists of known bad IPs from sites such as http://www.mtc.sri.com/live_data/attackers/ and http://intel.martincyber.com/feeds/ip, known bad domains from sites such as https://secure.mayhemiclabs.com/malhosts/malhosts.txt andhttps://zeustracker.abuse.ch/blocklist.php?download=domainblocklist.ArcOSI essentially is a Python script you run on a cron job that goes out to a bunch of sources scrapes these sites, runs a regex on them for IP addresses and domain names, then streams the data in CEF over syslog for processing and correlation.
- TweetLog – This something a colleague turned me onto from a company called MetaNet. The company is actually founded by some ex-ArcSight professional services guys we worked with in the past that are quite good. Essentially, TweetLog is a a real-time Twitter to syslog conversion tool.TweetLog converts Twitter messages (tweets) into syslog events. It’s built to follow individual Twitter accounts. The TweetLog service monitors tweets in (near) real-time and has an option to save them to a local file as well as forward them to a syslog server. It can be installed and ran as a service on Unix or Windows operating systems.MetaNet also provide an updated Twitter feed that even if you don’t want to use TweetLog is still a great data source. It can be reached under https://twitter.com/meta_net/feeds
- RepDV – Or Reputation Digital Vaccine by HP (formerly ArcSight). This is a commercial product that is licensed based on how many data sources you are collecting logs from in your ArcSight environment – so it could be pricey if you have like 1000 web servers or something. HP RepDV is a comprehensive database of malicious IP addresses and DNS entries.It helps identify and block access to “known bad” sites, reducing the risk of exposure for the organization. The HP has limited information – http://www.hpenterprisesecurity.com/products/hp-dvlabs/hp-reputation-digital-vaccine-repdv/ so I would suggest containing your ArcSight account rep. From what I have been told, it pulls intelligence from globally deployed HP Tipping-Point light-house attack sensors, world-wide HP TippingPoint IPS installations, third-party Malware, Web, and E-Mail Research, eSoft, SANS, Malware Domains List, and Sunbelt Border Patrol List. Produces some nice data such as below: