I'm Not Dead Yet

Truly, I'm not dead yet.  I realize it has been a long time since I last posted.  I'm working a new role now (new? Almost six years,) where I manage an information security team which encompasses our security operations center.  I have a great team, albeit a small team, located all around the world.

I thought I would revamp the blog as I deal with new issues and challenges in the "new" role.  What better way to reach out to many others who have solved some of these issues or can provide differing opinions and viewpoints to help shed light on potential paths forward.  I've found that writing (and asking) help alleviate the pressures and hopefully will be able to pick up some new ideas.

Those of you have have seen this blog in the past may notice new labels.  However, no posts or content will disappear.  And, none of the old labels will be deleted, unless I don't find any content associated with label.

HTTPS On!

This article from Google announced that Google is turning on https for blogspot.  And, as such, I have happily turned the setting to "on".  Good news readers, you can now read this blog over https.

Should you encounter any issues with formatting or functionality, please leave a comment.

Thanks.

Windows 10 and Citrix Netscaler VPN does not work

A quick post:
We have found that our users that have updated themselves to Windows 10 have effectively locked themselves out of the network, from remote locations.  It seems Windows 10 is not supported by our Citrix Netscaler VPNs.  Of course we put a note out to our worldwide users, but some "had" to update.  Hopefully there will be a patch or fix soon.

At the moment, we don't support Windows 10 in the infrastructure.  That doesn't stop some of our users from updating anyway.

Found a Windows 2003 Server - now to Remediate

I've always known that we have a handful of Windows 2003 servers.  And with the retiring of Windows 2003 on July 14th I started beating the drum towards migrating to newer (supported) servers.  When I last looked, I think that there was two or three servers left in existence.  Flash forward to today, and our weekly external vulnerability scan.  One of the scans picked up a Windows 2003 server in the publicly facing DMZ, and flagged the finding as critical.  I can support that, it's a pretty serious finding.  The scan also noted IIS 6.0, another finding.

After consulting with the Sysadmins, we found that there are only two applications being utilized on the server.  Off I went to query the application owners as to migrating the applications off the server to something supported.  Here's one of the responses:

I will speak with my management regarding making the migration of what's on there a priority to move. The wrinkle is that some of the code is old and needs to be rewritten/ported into the current project and that is an effort that I have yet to get permission to put the hours in on.

So for now, we need to leave it where it is and I'll try to get everything off of there as soon as possible.

Sigh.

Security education only goes so far.  The fact that there are exploits out there does nothing for the application owners...they're willing to take the risk.  I have their acceptance of risk in writing; but a lot of good that will do if/when something happens to the server.

Failed a Pentest...Stake in the Ground

I've been with the company a little over a month and a half.  I've run numerous gap analysis, and I know where we are deficient.  And some of it is  not good.  I've compared ourselves to the SANS Top 20, and again, it's not good.  Management wanted an internal pentest, to get a feel for the security posture.  We in IT wanted a good boutique pentesting company, but we were told to use the company that already audits the finance department.  Fortunately, these guys were good.

We failed the pentest, miserable.  Most of my guesses as to how it would happen came to pass.  And I'm ok with that.  Heck, they had domain admin in about a day.  There were some good surprises, and there were some good wins.  I'm good with it; as it confirmed most of what I have been raising to management for the past year.  The hope is that management will open their eyes and start making changes.

So mentally, I've put a stake in the ground.  I want to see how long it takes for any real change to take place.  I'm waiting to see when management starts mandating change in the form of implementation of controls in order to raise the security posture of the company.  Or, is management just checking a box that an audit was performed.

I'll update as controls start being implemented.

P.S.: I have to say, as a former auditor, it was interesting to experience the audit from the other side of the fence.  I was able to understand what the auditors were looking for and better able to answer their questions since I had been in their shoes.

2015 To Do: The Low-Hanging Fruit

I'm not going to try and recap this last year it's been great.  I know I've done good things and improved the security posture at the company as best I could.  Sure, there's more I could do, there's more I want to do, there's been battles won, and battles lost.

So, as a mental note, and to set a baseline, I'm outlining these mini-projects I want to get done as fast as possible.  I'll try to revisit this so I can see how long it took to complete these endeavors (and hoping that they get done.)  And this is not in any particular order.

1.  I'd like to get Two-Factor Authentication (2FA) on all the servers.  We use 2FA for VPN connections and it works well.  However, I would like to get it added to all of our production servers so we can (a) better track logins to these servers, and (b) add the extra authentication step to critical and production servers.  One challenge here will be leveraging our existing 2FA infrastructure and add it to servers.

2.  We have many proxy services employed in the network infrastructure.  Headquarters has a slew of them, depending on where network connections start and finish.  (Layers :-) )  When the headquarters (and data center) moved in October and November, our Zscaler connection was knocked off line.  This did not hamper headquarters much due to the other proxy services, but some of our branch offices rely on Zscaler as the primary proxy service.

3.  Our firewall solution is pretty robust - we have a lot of rules defined (that's a whole separate project.  Cleanup.)  The firewall has an IPS blade that receives signature updates from the vendor.  However, the network team has not implemented the signatures "because it's too hard / will block too much (!) / might cause a load on the firewall."  I want to come up with an automated solution where we can "auto-approve" most signatures.  For example, it would be great to come up with a policy where all Critical and High signatures get applied automatically.  Further, anything else that has a high confidence and low to medium impact we would apply as well.  The rest we can look at.  It would be a start at keeping the IPS in tune with current threats.

4.  Our GPO is used more for creating accounts and putting those accounts into business groups.  It is not really used to enforce security controls.  As such, there are a bunch of low-hanging fruit type controls we could implement without causing much pain.  Controls like locking screen savers, account lockout polices, and some password policies would be easy wins.

5.  We have an AV solution, it updates, and it appears to do it's job.  However, we don't have scheduled scans automatically turned on.  The users complain.  However, I suspect the AV will miss things without a scheduled scan to look.  Already, I've piloted turning on scheduled scans with a group to see what the real issues are.

6.  Our firewall and managed SIEM do a great job of alerting on known threats.  Our process to block some of those known threats is manual, though.  When we get an alert, we have to research the activity, then add the source address to a blocklist.  Manually.  There has to be a method to automagically block those sources on the first "malicious" event.  We need to turn this on.

7.  One of our FireEye appliances was taken offline due to the move.  We need to get this appliance back up and running.

I feel this list is simple enough; where completion of the items on the list will raise the security posture of the organization without many costs.  We'll see.

EMET 5.1 - Windows 7 64bit - IE 11

Our user machines are deployed with Windows 7 64bit and IE 11 installed.  I notice that when I go to sites that check the browser, the sites respond that "browsers less than version 8 are not supported" or words to that effect.  Usually, the browser works fine and there are no issues.  I have been trying to get Microsoft's EMET to work with this configuration since version 4.0; and I have not been successful.  Internet Explorer crashes with EMET running.

I read that a new version, 5.1, might fix this.  After setting up EMET, and tweaking it to my environment, I checked Internet Explorer, and it still crashes.  Are there fixes or workarounds for this issue?

Finding Users Who Use the Conference Room Computers as a Proxy to Surf

I received an interesting alert today, indicating that a host in a conference room was attempting to reach out to a site hosting an exploit kit.  This is not the first alert I have received on this machine, so I was a little puzzled.  So, I went to the machine in order to remediate; and noticed that the AV software on the machine had blocked the connection attempt.  I ran both AV and Malwarebytes to ensure nothing is found.

The machine was clean.

Just before I logged out, I noticed that Windows Update had run and needed to restart the computer.  As I clicked to restart, the warning notice that other people connected to the machine would lose their connection.  Hmmm....what's really going on here?

A little digging showed that you can find out who is/has logged into a machine via RDP by examining the Event Logs.  Open Event Viewer, and navigate to:

Applications and Services Log -> Microsoft -> Windows -> Terminal Services - LocalSessionManager.

There you will find events for who logged in, with what account, and from what source.

Now to go find some users......

(And yes, we need to fix our policy on conference room computers...but that's a battle for another day.)

Finding a Specific Microsoft Patch on a Host

After the Word (.rtf) 0-day was announced at the end of March, we turned on an alert to let us know when an .rtf file was delivered to the company.  Until the patch was applied, we actually blocked the incoming mail, inspected it, and if it was clean, we allowed it to reach its destination.  After the patch, we just alerted on the incoming mail. 

It's been a couple of months, and we are still getting the alerts.  Before I turned off the alerts, I wanted to ensure that the patch was on my host.  A quick script I ran to look for the specific patch was:

wmic qfe | find "KB2953095"

It seemed to work ok.

If there are better/easier ways to do this, leave a comment.

Somewhat off-topic....I can't believe the number of people that still send documents as .rtf.  Why not just use Word?  Or a text document?  The number of incoming .rtf documents was way higher than I would have guessed.  Most were resumes or travel booking documents.

SANS SIFT 3 and the Desktop Share

I had the new SIFT 3.0 downloaded for a while, but I haven't been using it as much as I would like.  I've been using the older 2.x version. One of the main reasons is that on the 2.x version of SIFT, there was a desktop shortcut that took me directly to a directory of the host OS.  This is missing in the 3.0 version of SIFT.  I fully admit, I don't know linux as well as I know Windows.

Quickly reading up on the issue, and I found that this mount to the guest OS should be found in mount_points/hgfs.  I had that directory, but nothing was populated there.  And, in the Virtual Machine Settings, I had the Shared Folders set to Always Enabled.  Still nothing.

On a reboot, I noticed that there was an update to VMWare Player.  I updated, and checked the mountpoints directory, but still nothing.  One last google suggested running vmware-config-tools.pl.

Sure enough, after answering the questions, that did the trick.  Now, in the mout_points/hgfs folder, I see a subfolder for "C".  Bingo.

Now I have to get used to Unity and finding what I used to be able to find in SANS SIFT 2.x.

If anyone else has tips on making that transition, feel free to leave advice in the comments.

Finding Inactive Accounts

The SANS Top 20 Controls has a control named Account Monitoring and Control.  Within that control is a Quick Win:  Ensure that systems automatically create a report on a daily basis that includes a list of locked-out accounts, disabled accounts, accounts with passwords that exceed the maximum password age, and accounts with passwords that never expire. This list should be sent to the associated system administrator in a secure fashion.

We don't have an automated report of those types of accounts, and quite frankly, we have very poor visibility into account control.  Coming from a DoD environment, I'm not used to having such lax controls.  Slowly, I'm starting to push the company forward, but it is taking time.

My first thought was to look at the inactive accounts.  I figured that these accounts would be low enough of the low-hanging fruit to start with, and here's how I have gone about finding them.

(Note: that I have created a master script that will do more than what this post details...I'm only describing inactive accounts at this time.)

1.  This command is in a batch file:  dsquery user -inactive 4 -limit 3000 > accountout.txt

Call the output file what you like.  The -inactive 4 parameter tells dsquery to look for accounts that have been inactive for at least four weeks.  I picked four to start with, as I realize that we have users that travel extensively.  My hope is that once we manage the output, I'll be able to lower that number.

2.  I took the output of the file, and copied it to Excel.  From there, I went to Data>Text-to-Columns in order to break up the data nicely.

3.  Column 2 seemed to be where I could differentiate between user and non-user accounts.  I filtered on just user accounts and copied that to a new sheet.

My results were staggering.  There are way too many accounts.  My next step is to find or create a process to validate that these accounts are a) legitimate, and b) truly inactive.  Spot checking a bunch of these users revealed users that are contractors.  And, if I have to guess, they are no longer with the company.  Prime targets to attack - which is why they should be disabled or deleted.

Once that's done, I'll need to automate the process and schedule it to run weekly or so.  As for locked-out, disabled, and password length checking...those will be added in time.

Letter to Management

This letter was posted today....and it could have been sent to our management.  (It's not.)  Many of the points echo exactly what is happening here.  I would say, the biggest excuse I hear is that management does not want to disrupt the corporate culture in implementing security controls.

I fear that a breach or severe incident will be the catalyst for change and implementing controls.  Yes, I've had many small wins, but there is lots to do.

humans.txt Appearing in our Firewall Logs

This morning I came in to work figuring I would continue working on analysis of our infrastructure for the Heartbleed bug.  We seem to be fine...this is a case where fortunately, we get lucky because we have been using such an old version of OpenSSL.  Which means we're probably vulnerable to a whole host of other vulnerabilities.  But, we appear to be not vulnerable to Heartbleed.  Which is a good thing.

Early, a co-worker came to me asking if I had seen the note from our DDoS mitigation provider.  It was the first such email to provide a source address for the attack.  The "attack" only lasted four minutes, and to me, was not much to worry about.  However, there was at least an indicator to look for in the logs.  I popped the source address into our firewall logs, and was presented 134 records back; all targeting various servers of ours. 

And here was the unique finding.  Every request string looked something like:

http://ourserver/some_bogus_directory/some_bogus_file.php?php121dir=http://www.google.com/humans.txt

I fully admit, I had never heard of a humans.txt file; I knew about robots.txt, but not humans.txt.  So I looked it up.  We don't use it.  Next, I fetched Google's human.txt file to see what was in there.  Nothing untoward.

The best I can come up with is that this is some kind of remote file inclusion attack and the attacker is looking for vulnerable php servers.

I found a great site that had a little more info here, but their mitigation was in using .htaccess; we use our firewall.  I did not find much more information, so anyone that wants to shed a little more light on the subject, feel free to leave a comment.

Sophos Antivirus and EMET

In looking for mitigations for the recently announced Microsoft Word 0-day, I decided to install EMET on both my desktop and my laptop.  I fully admit, I'm not an EMET guru, nor do I know a lot about it.  I have found many directions for EMET (a good one here) so installation was a bit of a breeze.  However, tweaking it is another story.

First, Firefox 28 wouldn't start.  So, I had to tweak the application settings for Firefox to find out which particular protection was preventing it from starting up.  (Turns out, it was ROP.)

Then, upon turning EMET loose, I received two "Quarantine Announcements" from our Sophos Antivirus.  The notice was for a buffer overflow in IE and Acrobat reader.  From my analysis, the best I can tell is that Sophos saw EMET protecting those applications and didn't know how to report it.  I asked our Sophos administrator if he had heard anything about Sophos and EMET, but he didn't know what EMET was.  I authorized the activity in Sophos, and rebooted a couple of times to see if Sophos would report the activity each time I booted up.  So far, so good.

If I find out exactly how the buffer overflow was caught by Sophos, I'll update this post.

Python Links Updated

Just a quick note that I've updated my list of Learning Python resources after reading Harlan's great post.  My original (and now updated) post (and list) can be found here.

Getting Started with Security Onion

After getting alerts from our DoS protection company that are vague, one of the network engineers and myself decided we needed to gain more visibility into the network.  We want to better understand these events and make a decision as to whether or not they are truly incidents.  Further, after we get notification of an event, we want to find the traffic to study it.  Enter Security Onion.  This tool is awesome, as we can run Snort, shoot the output to Snorby, and capture the data as well.  We're pretty sure that we have a box capable of running Security Onion, it's more a matter of how much data we want to keep.  Right now, we have a 1+ terabyte drive doing the heavy lifting.  We're just barely making it before the job to purge runs.

Our first shot at getting it up and running was fairly successful.  Data is flowing, we saw some alerts.  Next on the agenda was to start tuning it such that we are not drinking from the fire hose.

And, now we've broken Security Onion.  We're not sure where yet.  Events are coming in.  Our sensor NIC has packets traversing it.  However, there's nothing showing up in Snorby.  So, on to more trouble shooting.  Fortunately, this Security Onion server is not production-ready.  We knew going in that we would have much tuning before we could start truly relying on the output in a production environment.  The next step is to figure out what broke down and see what we can get back.

Finding hostnames on a Subnet

We have offices all over the world.  What I came to learn today is that there are two countries where we have very poor visibility into our own corporate networks.  To the point that I suspect that they are not managed very well, if at all.  I know I chase down malware in a couple of the networks on a daily basis. 

One of our admins asked me if there is a way that we can get all of the hostnames on one of those subnets that we don't have much visibility to.  NMAP would have worked well, but I wanted to come up with a command that I could have had a non-technical person run and send me the output. So, using a little Command Line Kung-Fu, I came up with:

for /L %A in (0 1 255) do nbtstat -A "XXX.XX.XXX.%A">>hosts.txt

Substitute your subnet for the Xs in that command.

It worked like a champ.  I suspect that there is an easier way to do this, but this worked easy enough.

Hunting for Zeus Throughout the Network

I average finding a little more than one Zeus infection a day.  I know the reasons.  The root causes are there are some major security controls missing from the environment due to culture.  Adding those controls is a challenge and is a long-term strategy.  We are in the infancy of a Security Awareness campaign that is just starting to teach people the dangers of clicking on links in Spam or falling for phishing.  Occasionally, the FireEye sensor would alert to someone clicking a Zeus link.  I suspect more click the links than I am aware.

Using the Check Point's SmartLog, I've worked up a little query to help me spot some of the big outbreaks.  I grabbed the domains from the ZeusTracker, and built a mini-query (which I then pasted in the query bar.) 

dest:(domain or domain or domain or bizserviceszero.com or ....)

Periodically, I'll check the domains on ZeusTracker and run a diff to see what enters the list and what gets removed.  I know that there are better ways to do this, and I'd love to implement  some of those methods.  High on my list is adding a Snort box, or even SecurityOnion.

A small win for the day, but at least I can find these machines.  Hopefully, when I've built up some metrics, I can support changing the environment, and use the number of infections cleaned up as the driver.

RSA Conference: Friday

I woke up on Friday to cloudy weather.  With an afternoon flight, I was on the fence with going to hear more talks.  I figured I would wait until after a shower.  Upon getting out of the shower, it was POURING out.  I'm glad California was getting the rain it needed, but...I didn't want to walk in it.  However, after getting dressed, it had stopped raining, and actually looked like it was lightening up. I decided to chance it for one more talk.  I probably could have gone to two...but there was no way I wanted it to be close at the airport.

The talk I went to was:

• Operation Olympic Games is the Tom Clancy Spy Story that Changed Everything - by Rick Howard.  Wow, this was a great talk. I'm looking at my notes, and I see that I stopped somewhere after the first ten or fifteen minutes.  It was that good and engaging.  Rather than tell the technical story of Stuxnet, this talk discussed the history of the operations and the planning that went into it.  Further, Rick put forth some interesting theories that certainly have merit.

And with that I bid the conference goodbye. I fully admit that I wasn't sure what I would get out of it, and expectations were exceeded.  I've written the dates down for next year, and we'll see what happens.  As I write this, I'm starting to plan on how I'll use and implement some of the notes and ideas that were generated at the show.

RSA Conference 2014: Thursday

I woke up and it was absolutely pouring out, I thought it was going to be a repeat of yesterday.  Fortunately, by the time I started walking down the hill towards the Moscone Center, it had stopped raining.  And, while eating breakfast I noticed the sun coming out, a welcome addition.  Breakfast is served 7-8 in the morning.  The first conference I had scheduled was 9:20, but I thought to myself, I don't want to waste the time, so I decided on an 8:00 talk.  I'm glad I did.  Here's the talks I went to today:

• Cloud Ninja: Catch Me If You Can - by Rob Ragan and Oscar Salazar.  Initially, I had this time slot open but at the last minute, I decided to pick a talk and go.  I'm glad I did.  This talk was awesome.  Initially, I thought it might be neat to hear a session with a little offense to it, seeing as how I mostly focus on defensive security.  But, as the talk focused on (ab)using free trials of company's software to build a botnet, I realized that there were dire implications for the company where I work.  This was a great talk that gave me information to go home and battle the developers.
• Keeping Up with the Joneses: How Does Your Insider Threat Program Stack Up - by Dawn Cappelli and Randall Trzeciak.  Probably of all the talks I scheduled myself to see, this was number one.  I have their book, so it was great to hear Dawn and Randall talk.  Of course they backed up their research with plenty of numbers and examples.  They gave great advice on building and working an Insider Threat program.
• The Future of Exploits, Developing Hidden C&C and Kittens by James Lyne. I picked this talk as I wanted to hear a talk by one of our company's vendors and I suspected it might get a little deep.  It didn't get too deep, and I'll tell you, I've never laughed so hard in a conference talk.  A great talk, kept light, with lots of great information.  And, now I've learned a great little story to explain buffer overflows.

I did not attend the Keynote talks today, instead I took a walk up to Fisherman's Wharf to see the Rock, seals, and a tour of the USS Pampanito.

I did attend the Codebreakers Bash, which was really well done.  They gave out these blinking LEDs, and now it is my hotel room has become a disco.  I'll have to cover it before going to sleep.

I fly out tomorrow, in the afternoon.  So, I'm on the fence with going to a talk tomorrow.  I'm tempted, to go see one more.  Probably the decision will be made by what time I get up. I will miss the keynotes tomorrow, and that means missing Stephen Colbert.  But I think I'll be ready to get on a plane.