Putting a disaster recovery plan into practice

As a DoD auditor, I hammer DoD sites and entities for not having or not having adequate disaster recovery plans.  I got to put words to practice in my own life with the impact of hurricane Sandy directly impacting me.  In light of what happened, we are revamping our procedures and amending the plans that we already had.

For example, we had planned for wind damage, and the possibility of a power outage.  We didn't plan on evacuating...data from previous storms didn't indicate that we might have to worry about rising water.  However, with the storm picking up steam, hitting us three hours earlier than anticipated and during a high tide; the stream around us was 13' above normal.  Water poured all around the house, which led to a prompt evacuation.

So, with a new threat to add to the threat model, we are revamping our plan.  We are including many of the lessons learned from this event so that we are not caught off guard in the future.  Further, we're keeping the plan in a central location so that everyone can benefit from seeing it and putting it into place.

It would figure that we would not get a hurricane until the very end of hurricane season.  And we know that Winter is just beginning....we're bound to have a snow/ice event.

Is it me, or is the TSA getting stricter on what we can carry?

After this trip, I will have been testing two of the last three weeks. (And I've already got a trip planned in March.) But, on the last trip, and on this trip, I've had my laptop bag given the extra once-over by TSA. On the last trip, the TSA rep in Kansas City took my bag apart and let me know the problem was with my cable tester. I've traveled many times with the cable tester, and only now did it cause a problem. When I asked what the (specific) problem was, I was told that "it contained a scary image on the x-ray machine."

On this most recent trip, it was the hub that caused the commotion. I didn't get an explanation; I didn't ask. As an incident responder at heart, I like to have everything with me that I might possibly need. The lesson learned is that I'm going to have to check most of the tools that I keep in my bag. I've always packed my tools (snips, screwdrivers, etc.) but it looks like I'll be packing and checking more of the tools in my bag. (Truth be told, I don't mind shedding the pounds.)

My question is, is the TSA getting stricter, or have I just gotten lucky in the past.

Getting into a Gold Disk-locked laptop and scanning a laptop not set up with network connectivity

Back from testing. It was a packed couple of days, and we worked in a pretty tough environment. Anyway, we had two sets of laptops that Retina could not scan, as it couldn't connect to the registry; even when we had the correct admin user name and password.

In the first case, we needed to scan two laptops that were not connected to the network. We had crossover cables, and we tried running the scans; each time though, we couldn't connect to the registry. My co-worker came up with a good trick. Because the laptops were in a workgroup, we were able to run the network setup. From there we ran the wizard, picked home network, but didn't actually set up anything. The process installs and starts the "server" service which allows the sharing of the drive. After that, we were able to run our scan and connect to the registry.

The second case presented an interesting problem. Again, we couldn't connect to the registry. However, this time, a little investigating turned up the cause. The client had run Gold Disk on the machine, and immediately clicked "remediate" when Gold Disk was finished. Yep, they locked up the box something good. Logging in to the machine from the network was not possible. To fix this, we went to Control Panel -> Administrative Tools -> Local Policies -> Security Options. From there, allow network login access.

Web Application Testing

Swamped. That's what I've been. I can't believe my last post was February; mid to late February at that. I've finished one engagement, and I'm in the process of writing that up. And, I've been thrown onto another engagement. This one's big, and of course has an end date of early May. The funny thing is, once we're done testing the system and writing the documentation, it's a 30-60 day wait for the decision on an ATO. We have almost 100 systems/applications to test. That said, we don't have enough time.

I remember this happening when I was a full-blown project manager working in the private sector. There would be some regulatory announcement that the company would have to adhere to. Instead of figuring out the requirements, figuring out the estimates, and doing the work; we worked backwards. Here's our end date....what are the milestones and when do they have to occur in order to get there. I'm finding the government is worse.

Anyway, I've been getting acquainted with NTOSpider, a web application vulnerability tool. Because of the crush to get this project done, we've already started testing. The PMs just gave us URLs, not system owners. We can't find anyone to own up to the systems, and, when we try, we get our hand slapped by the PMs. Of course, today, an app I was testing was a help-desk type of app. Every submission of one of the forms generated an email. Hundreds of them. Probably more. So, now I'm trying to dig into NTOSpider to see what I can learn in order to fine tune our testing.

Send the Interview Questions to the Client, pre-test

I just had a client ask for some clarifying information regarding non-technical controls. It seems that the documentation did not fully enumerate those results. In order to ensure I have that information before leaving the client site, I will ensure the clients have the interview questions before I arrive. I believe it will allow the interview to run quicker in that the client will already know the questions. Also, if they've filled out answers, I'll have a hard copy I can take back to accompany what I've documented myself.

I'll see how this goes shortly, as my next testing engagement is scheduled for a couple of weeks.

SQL Server scans

A new lesson learned. At least, I'm filing it that way so I can jog my memory for future testing engagements.

We tested at a client the other week that claimed to have one Oracle database, running on top of a Windows 2003 server. It turns out that they had another Oracle database, sitting on a Solaris machine. (The IT department didn't know about the database because they didn't administer the machine....a whole other issue.) That wasn't such a big deal, as we had the scripts to test the database with us. However, while a co-worker was interviewing the DBA, he happened to see a MS SQL Server instance on the DBA's monitor. When we got back to the office, I poured through the vulnerability scans looking for a sql server. I found five. Three instances were found on client XP workstations. And, if I had to guess, those instances probably came bundled with specific software that was installed. A whole other issue for these networks. However, I found two instances residing in the data center on servers located there. Knowing this client, I think they were just forgotten, or not included because the databases were not part of a web application. But, they definitely should have been scanned, and it was noted in our initial documentation.

So, after each testing engagement, I'm searching the vulnerability scans for SQL Servers (of any type, for databases not mentioned to us;) both in the datacenter and on the client LAN. And, I'll probably do this early, so we can scan/test the databases the next day.

All testing will be much more stringent now

When testing a site, we either are testing with the intent of writing an initial security assessment report; or, final testing to complete a DIACAP package. For one of my first engagements, we were testing for an initial assessment. So, we grab data from a representative sample of like machines. However, just this week, most likely due to external politics that I am not privy to, the decision was to create a final DIACAP package from the data collected. Obviously, the customer is not going to get the best picture of their security posture. And, there are highly important issues that will get reported instead of fixed with an initial security report.

It's been highly frustrating for me, to say the least.

However, the lesson learned is that from now on, I will test every system as if I am testing for a final DIACAP package, even if the outcome is an initial report.

When to perform the interview during an accreditation

Normally, when accrediting a system, there is a team of us security warriors; probably performing a myriad of tasks. The interview of the Sys Admins occurs when any one of us has a spare hour or two to ask the "non-technical" questions and go over documentation and process. However, for the engagement I am currently working, my partner suggested to our client that we perform the interview FIRST in order to get the interview (and pain) out of the way and allow us to test the systems during the rest of the engagement.

One lesson I think I've learned from this move: By performing the interview FIRST, we find some issues/areas where we may want to take a closer look. The customer may have inadvertently said something that gives us reason to look at a particular issue. Or, they may say something that leads to a finding we might never have found. And, if the interview is conducted late in the engagement, there might not be time to further investigate.

I'll know soon enough, as we cast our eye about the network and systems starting tomorrow.

Be extremely detailed when taking notes

Another lesson learned here. I'm going through notes that I took on some of the systems we worked on. And, I'm finding that my notes are not as detailed as I would have liked. For example, I wrote at one point: "starting SRR script on first Solaris machine." However, it would have been better if I documented the full version number of the operating system.

Going forward, I want to try to remember to write down: machine name, IP address, OS, and version number, and specifically what is being done.

Retina and scanning network sizes

I just got back from my latest testing engagement. Overall, it was a good trip. The site was not as prepared for our testing as they thought they were. And, as such, they were not as happy with our results. Suffice to say, they have some work to do. However, it seems that the team hasn't been together that long, so there is much upside, and I'm sure they'll come together.

And again, for the second straight trip, this organization had "problem children" that seemed to flaunt the fact that they were not going to play by the established rules. Of course, it will all come out in the documentation.

The lesson learned from this trip: break up the Retina network scanning of clients into subnets. There are over 500 hosts in my .rdt file. It's my fault, I let the IASO perform the scan (although I was watching over his shoulder.) But, that file is huge. And it's impossible to load into Retina. It takes forever to load and produce reports. Besides the size advantage, scanning by subnet will aid in keeping the files manageable.

I have another trip scheduled in two weeks, so I'll get to put it all into practice.

Some lessons learned from my last trip

I just got back from my latest testing trip and I've come up with a small list of things that I learned from the trip. For this trip, there were a couple of items I didn't bring, and need to remember for future trips.

Wire-bound notebook - I brought my planner, and figured that would be enough. A) The planner is just too big to carry around. B) I was carrying around more than I needed. C) A notebook will be great as I can date the pages (and number them) for each testing engagement. D) The notebook will be smaller (and lighter) than the planner.

Site Physical Security Checklist - While this is loaded on our test laptops, there was no easy way (in this particular case) to get access to a printer. I need to remember to print this out ahead of time so I can use it on site.

Notebook mouse - wired or wireless, doesn't matter. My wrists were killing me after using that pointer above the B. And the touchpads were horrible. A USB travel mouse will go along way.

Lesson learned: bring a different browser

Out on a case today, I ran into a situation where I couldn't get to the internet. Something had overtaken IE. I was really wishing for Firefox at the time, but I had no way to get it. (And I couldn't remember the FTP commands to grab it.) Now, I've downloaded the .exe to my USB drive, so I should be ok when I need the alternate browser in the future.

Lesson Learned: Always mention when you are going to analyze a machine

It's been an interesting week. People have been leaving the company in record amounts.

The latest occurred yesterday. The manager went to his boss, gave his resignation, said he was going for coffee and would be right back. He hasn't returned yet. At least as far as I've been told. My co-worker disabled the network account and email. He went down to the machine and uploaded to the network any files that were on the C: drive, thereby not being backed up.

We don't have a policy on what to do when a person leaves the company, willfully or not. I've tried. HR doesn't want the extra work.

Later in the afternoon, I figured I would take a look at the ex-employee's computer; looking for deleted files, pictures that shouldn't be, or anything else that shouldn't be on the company computer. So, later in the afternoon, having a few minutes to spare, I head down to the ex-employee's computer; wip out Helix, and start analyzing. I really didn't expect to find anything. I was sidetracked on the way, so I didn't mention to anyone where I was going.

I went to look at IE history, but accidentally hit the button for Nirsoft's Protected Storage Pass View. Well, the Trend Micro client installed on the machine picked it up as "hackerware." A note would be sent to the administrators. About ten minutes later, I there's a knock at the office door, and there standing outside is my co-worker and my boss. I quickly explained what I was doing. They were there because they thought the manager who left had come back and working maliciously on the computer. All was soon well. Important lesson learned, though. Let someone know what you are doing so as not to falsely set off alarms.