Pulling timelines from unsupported filesystems.

The conference season is closing in and workload is finally easing up enough to put together a couple of blog posts. This one has been in draft status for months.

If you use timelines for investigative purposes as much as I do, you have no doubt run into occasions where fls does not support the filesystem you are working with.  I have been working a lot of E-commerce lately and bumping heads with several different variants of Linux filesystems.

Here's the list of filesystems supported by fls:
 

Notice that ext4, xfs and several others are not in the list.

I had to go digging for a way to do this for an xfs filesystem  that I was working with. Fortunately, I have a decent library of books on filesystems, forensics, security, etc.. I remembered reading about using the find command to generate timeline data and I was able to find it eventually in "Incident Response & Computer Forensics." (Mandia, Prosise & Pepe) Thanks guys!

They described the process of using the "find" command to generate the file but not all the steps to getting there.




The steps:

1.  mount your filesystem read-only. Here's the command on  a *nix box.  (this can be done for some filesystems in Windows with a range of different utilities)

mount –t xfs –o ro,loop,noexec /media/USB/image.001 /mnt/apachesdc

2. Change directories into your newly mounted fs:

 cd /mnt/apachesdc

3. Run your command and output it to a file for later use:

Find . –printf  ‘%T+ , %A+ , %C+ , %p \r’ > /path/to/output_file.csv

This may require a little massaging to get it to work with your particular distro. The man pages are your friend.

This find command will output Mtime, Atime, Ctime, full path and filename, as well as end each line with a carriage return. The commas in-between add a common delimiter, for later use with excel.

Using the capital T, A, and C with the find command allows you to specify the time/date output format you want. In this case I wanted  yy-mm-dd+hh:mm:ss . This format is natively understood by excel and makes the spreadsheet much easier to sort.

Open the csv file with excel and set “comma” as the delimiter. It will neatly organize each row with MAC and filename.  Add a row at the top and label each field, lock that row, and you have a spreadsheet that is sort able by M,A,C, or filename. 

You don’t get all the extras that you get from using fls and mactime. No GUID, *, *. But what you do get is a very functional timeline for multiple unsupported filesystems.

I've decided to write a talk that focuses a little heavier on E-commerce this year. My co-workers at Trustwave have the POS malware well covered and DEFCON/SECTOR were fantastic events last year.

My working title is "Ask me about hunting carders for fun and profit" and I plan to include some stupid hacker tricks, some malware, and a quick look at data reduction using the command line.

Stay tuned for a series of posts on some E-comm breaches. I have drafts in work.


On a more personal and non-technical note, my whole family has been involved with the St. Baldrick's charity for 5 years now. We co-organize a head shaving event to raise money for childhood cancer research. My little man has been shaving his head (totally voluntarily) since he was 4 years old. Do me a favor and take a look at the charity, see if there are any local events you can support or drop a couple bucks in one of our donation boxes.

My donation page.
Max's donation page.


Thanks,
Grayson

MAC(b) Daddy at SecTor

I'm proud to announce that I was invited to deliver I'm your MAC(b) Daddy at SecTor 2011 as well as take part in a full day of training for the Royal Canadian Mounted Police.  If you haven't heard about SecTor, read here.  It's Canada's largest security conference and is described as "The Canadian DEFCON"



I feel honored to have my talk accepted and I'm looking forward to meeting new peeps.

Hope to see you there!

I'm your MAC(b) Daddy at DEFCON 19

It's hard for me to believe that I haven't updated this blog sine March 26th. The last 5 months may have been the busiest of my entire life.  Three members of my team(including me) worked a nationwide breach on over 80 locations all using the same Point of Sale software.  I also submitted to, and was accepted for two conferences. DEFCON 19 and SecTOR in Toronto.  I have been working diligiently to make sure my presentation slides were clear and up to date and to make sure I was ready to get up there and speak in front of hundreds of people. I'm making excuses for myself here, the better thing is probably just to get on with it.

For a little background on Timestomping and why attackers are doing it, see Chris's post "Timestomping is for Suckers".

I presented a talk on Supertimelines and identifying anti-forensics at DEFCON this year.  Aside from some minor issues trying to pull off a live demo, the talk went pretty well.  I had the unceremonious duty of sharing a time slot with Dan Kaminsky so I’m very happy that I managed to fill 2/3 of the room.  I’ve already started receiving a number of questions, links to others research and a number of other queries related to MAC(b) Daddy.  Keep them coming, I’m more than happy to help out where I can.

I am presenting MAC(b) Daddy again at SECTor in October. Once that conference is over I will post the full content here on my blog.

The first communication I received was a link to a blog that was exploring different Timestomping methods using the Windows Powershell and the timestomp utility I mentioned in my talk.  There is some great research and concise info about manipulating timestamps.  The link was sent to me as a “Hey, this guy was able to modify the $MFT, and you said that couldn’t be done”   Well, sort of. What I’m saying is that the $File_Name attribute can’t be modified by anything known, the blog above proves my point. Manipulation of the $Standard_Info attribute is, dare I say, easy?, at this point.  The second set of attributes in the $MFT is still untouched by timestomp, powershell, perl scripts….anything.  By comparing the two you can see the changes made to the system by these measures.

Directly after the talk I was approached by two Chinese gentleman that had a number of questions about trying to modify the $MFT with a kernel mode driver. I asked, “Why, what are you working on?” They replied with a smile and said “nothing”.  This is DEFCON after all.  This is certainly an interesting project but one that would require extreme caution. Modifying the $MFT on the fly could be extremely detrimental to a system. Move the file table entries by one block and you just turned your system into a brick.   Not to mention that you would gave to query a protected file, make the change, leave the sequence number undisturbed and release the $MFT before the system itself tried to write to it again. Past experience suggests that you have 10 – 15 milliseconds to perform these actions.

Third, and maybe the most fulfilling for me, I was contacted by another forensicator who is working on a homegrown utility for parsing the $MFT and auto-comparing the entries for time anomalies.  This functionality is included in the latest version of Log2Timeline as well. I have not used this particular module yet but I plan to in the next couple of weeks. His questions related to some anomalies in a number of the core OS files (like ntldr).  I am by no means an expert here, but the way I understand the anomalies in these files follows:  One of the only ways to actually pull off any “manipulation” of the $F_N attribute is to create a file on a second volume (D:\), modify the $S_I timestamps, and then move that file to the main volume (C:\).  In this case the M attribute of $F_N will match the M attribute of the moved file.  The same does not hold true for the B attribute, which creates a whole other anomaly in of itself.  When we are talking about these core OS files, I think there are two things going on here.

1)   The system isn’t a system yet, it has not gotten to the point where the system time has been determined. This is the same reason that you see registry entries in a supertimeline start in 1969 and 1970. The system has no baseline to set those registry write times to and the possibility exists for the same issue with the $MFT.

2)   These files are not being created out of thin air, they are being moved from another volume (The install CD/DVD) and some of the timestamps from when this code was written is being maintained.

I promised the crowd to start updating my blog more frequently in support of MAC(b) Daddy. So here I am, a man of my word.

More as the questions and comments flow in.

Windows Registry Forensics-Review

I read Harlan Carvey's "Windows Registry Forensics" on a flight to Florida last week so I thought I'd write up a little review.

If you haven't already read "Windows Forensic Analysis" I highly recommend you do so.

First of all I have a tremendous amount of respect for someone who is willing to put their thoughts to paper when it comes to digital forensics. It's a very difficult topic because things are always evolving. It also takes a tremendous amount of time to write a book,  I'm trying to help a buddy with just a few chapters in a book he started and it's extremely difficult.

I will admit that I was expecting a little more book when I first purchased it, the registry is such a large piece of the Windows OS that I really thought the book would be encyclopedic. That being said, I was not disappointed  by the content.

The book is layed out in 4 chapters: Analysis, Tools, Case Studies:System and Case Studies: User tracking.

The Analysis chapter covers the binary structure of the registry as well as it's main purpose to the operating system and to the users. A considerable amount of this section was review for me (ten years of sysadmin work) but I've never read anything that tears down into the physical on-disk structure of the registry at the lowest level. Harlan obviously spent some time in this section tearing the registry down to it's nuts and bolts.

The tools chapter: If you don't read Harlan's blog or keep up with who is doing what in the industry you'll be expecting him to spend 50 pages talking about EnCase. Instead he uses the chapter to talk about a myriad of other tools which are just as useful if not more so than EnCase. He spends a fair amount of time explaining his own Perl tool "Regripper" and how it came to be as well as encouraging readers to develop their own plugins for regripper. I personally use regripper on every case I work so I know how useful it is and have spent a fair amount of time trying to figure out how it odes what it does. The book helped explain a little bit of the "behind the curtain" thought process that went into it's design.


Case Studies: System. Here's where the book starts to really pick up. 72 pages of nuts and bolts and "Here's why I've spent the last 150 pages explaining all this crap to you."  I read this section twice.  The registry contains so much information about the state of a given system that it is imperative a good investigator knows what they are looking at, what is normal, and why. After reading Harlan's case studies I have a better understanding of the pieces I already knew about and some insight into other chunks of the registry that have never caught my eye. Excellent chapter.

Case Studies: Tracking user activity.  The most useful chapter in the book! I am familiar with registry artifacts found during an investigation. For the most part I know how to use those artifacts to forward or disprove a theory. Even with that knowledge this section caught me off guard with how little I really know. Especially good was the dissection of all the areas of the registry that can be used for malware persistence and the write up on "shellbags". I have run across these artifacts a dozen or more times in my supertimelines but never payed them all that much attention, I knew that they represented user activity of some kind, but it didn't seem related to any type of malicious activity. Now I know that I was correct in my assumption but I also know that those shellbags are definitive proof of an interactive session.

All in all, I enjoyed reading the book. Harlan keeps it personable while maintaining an air of technicality. So many books like this are so dry that they can't be read for more than 15 minutes. Not so in this case. I have absolutely no regrets on the purchase price and just like "Windows Forensic Analysis" I will be referring to this book for years to come and I'm glad I have it as a resource.

PCAnywhere or "Here Hacker, Hacker"

Let me start by apologizing to anyone that was enjoying fresh, regular content from my blog. I haven't written a new post in several months

 I am not without reason. Starting a new job and getting used to the way things are done on a new team is time consuming and can require a lot of focus.  A wise man once said: "If you don't know what you're doing, do it neatly"  It doesn't fit entirely, but the learning curve has been steep, and my first few cases were worked through slowly, carefully and methodically.

This is not to say that my next cases will be worked any less carefully, but I've found you develop a rhythm in an investigation and you have to let the evidence lead you down the path to the answer.

Case in point:

I have already had a number of POS breach cases where PCAnywhere was the point of intrusion. I will focus on 2 for the purpose of this blog.  These 2 cases were similar in that they were both running the same type of POS software and it was set up by the same integrator.

PCAnywhere is a 2 part remote administration utility that has been around for a number of years. I don't know exactly when it came out but I remember running into it as a system administrator at least 7 years ago. It has gone through a number of version upgrades over the years and the latest stable version is v 12.5.  The 2 parts are the client, which is usually used by the remote administrator to make a connection to a remote PC or server for maintenance. The second part is the listener which (obviously) resides on the client machine that is to be remotely managed. Here's a list of the versions and the port usage (by version) from Symantec:

I have used numerous pieces of PCAnywhere evidence in my cases. 

The first is the connection logs. By default they are located in %SystemRoot%\Program Files\PCAnywhere and have a .pl9 extension. These logs are extremely configurable in the software itself.  They are easily viewed by using a registered copy of PCAnywhere or by downloading a trial from Symantec. These logs could contain anything from a basic timestamp and a connection attempt ,to full Windows Event Logging and verbose connection details to include connecting IP, user id info, files transferred, etc. For a full list of logging options, see this site.  Obviously these logs are very useful to an investigation. Unfortunately,  they are rarely verbose and in many cases I have found that logging is not even enabled. 

I have a problem with this. ADMINISTRATORS: IF YOU ARE GOING TO OPEN A GAPING HOLE INTO YOUR CLIENT"S ENVIRONMENT, AT LEAST HAVE THE DECENCY TO PUT IN A STRONG PASSWORD AND SWITCH ON THE LOGS!  Yer killin' me. You're also costing your clients tens of thousands of dollars.

There are also a number of vulnerabilities related to the older versions of PCAnywhere. Search the CERT website to read further on them all.  US-CERT Site.  One of these older vulnerabilities allowed you to pass the listener a message that says your login type is "0" Which allows you to log in with full admin credentials using a null username and password. That's Kwality there.

A bit more in depth is a file that I have found extremely useful that you probably don't know about. When you log in using the graphical interface a file called aw.swp is generated. This file is a cachefile that is created to speed up Windows operation.  It is used every time you use the remote desktop via PCAnywhere. This leaves a nice trail of birth, modified and access times in the system timeline. I even found evidence of card-holder data being moved in the aw.swp file. My assumption is that the built-in PCAnywhere file transfer button was used to move a file containing captured data. No one at Symantec ever contacted me back on this issue.  

Nothing ground-breaking here. It just goes to show that some of the most common applications leave behind some very definitive evidence. Sometimes you just have to dig around a little bit for an explanation of why.

I'm putting the finishing touches on a presentation for this years round of conferences. It's called "I'm your MAC(b) Daddy" and focuses on using super timelines to solve breach cases. I will post it here if I don't get accepted. I also have a post on generating timelines for filesystems that are not supported by fls and mactime. 

New updates soon. Thanks for stopping by.

"Perl's" of Wisdom

I spent the afternoon prepping up a laptop to take with me when I work cases on the road. If you've read some of  my earlier posts you know that I really like open source tools.

Regripper, The Sleuth Kit, UnxUtils, F-Response(not open source but worth every penny) and Imdisk are just a few of the indispensable tools that I think you should be using if you're serious about forensics. This is not to say that you have to jump off the deep end and run everything from a linux box. I'm not afraid to admit that I like to use Windows on my laptops. I'm not even afraid to admit that I like Windows XP.  

Here's what I don't like:

I don't like switching back and forth between machines to accomplish basic tasks, even if it's just popping open a VM and minimizing it again. Lame, hate it, I want  to be able to do everything from one machine. Why? I guess I'm just lazy and stuff. I don't want to bring 4 laptops with me to work a case. But I do want to be able to solve it while I'm on site.

So here we go,  If you want to crack off a timeline while those disks are imaging, I only know one way. F-repsonse and  Perl scripts.  F-response lets you deploy a tiny little client to each workstation, terminal, server or whatever it is you want to image, and suck all that data down to a single point. This is infinitely easier than attaching a hard drive to every machine in an environment, and it gets better.

The extra added bonus to F-response is that it offers up a live, read-only, environment to play with while that painfully slow image is running. 

It really is this easy:

Load up F-response listener 

Deploy client to nearly any OS in existence

Push start

Run your favorite imaging client (I like FTK lite)

Image hard drives from a single point on the listener.

Here's the bonus.  On the same machine that functions as the F-response listener, you have a mapped drive to the same physical disks that you are imaging. They show up as a regular mapped drive! E:, F:, G:  all read-only and there for you to get your forensics on.

Want a sneak peak at your timeline? Bam! 

fls -m 'C:\' -f ntfs \\.\: > bodyfile

Want to do a little password crack-a-lackin? Done! Use Ftk-Lite and extract that SAM hive.

Drop those hashes into Ophcrack and watch those default passwords appear before your very eyes.

This is not to say it's all a cakewalk. Thanks to Harlan there are some awesome perl scripts floating around the internet that will help you. Forcing these scripts to work on a Windows box can be a chore. Trust me, I just spent half my day proving it.

Case in point: regtime.pl  

This is Harlan's superfly, TNT perl script that shreds all of the registry hives(including NTUSER.dat)  and spits them out in standard bodyfile format.  (go back a few months and see my post on super timelines for use)

 The only 2 places I know of to get regtime.pl from are Harlan's email and the SIFT workstation. I didn't want to bug my mentor's mentor too much so I chose the SIFT download. 1.5 GB's later I had all the contents of /usr/local/src/windows-perl saved to my desktop.

Easy right? Install active-Perl from the website and go? 

Wrong.

BONK!

No biggie, I'm missing a perl dependency right?  BONK!
There's something wrong with the script? BONK!
Something got screwed up when I copied it over from Linux? BONK!
My Perl environment is not the polished marble that Harlan's is?  Well, maybe....   BONK!


WTH is going on here? Thought 1 turns out to be correct. This is a dependency issue. Even after you've been through your environment 10 times and all your dependencies look sweet, there's the registry.pm file in the right path and  you've contacted the developer because you've ruled out everything else. (sorry Harlan, I owe you  expensive scotch when we finally meet).  It's really very clear in all of it's perly non-clarity.   It can't locate Parse/Win32Registry.pm and this is clearly a misplaced file in a directory path right?  Not exactly. 


Try this.
From the command line type "ppm" for the perl package manager, go to view and check "all packages".  Glory, Glory, Hallelujah.  There's a package called Parse-Win32Registry. Right click and install that bad boy.

Now when you try that mystical regtime.pl command it coughs up a perfect bodyfile.

Coders are a bubble off; I'm telling you.


Next trick:  Path variables.

The command "dir" sucks, "ls" does not

ls, grep, cat, and a whole host of others are available in the unxutils package. Go downlaod it and install it and while you're on the web grab strings and grep.

Once you've got all these ported apps installed you don't want to cd into a specific directory to use them right?  You want to be able to use the command like you're the one in control.

Editing your $PATH is the answer.

Right click  my computer and go to properties--go to "advanced"--"environment variables"

Highlight "PATH" and click edit.  Use a semicolon as a separator and add the full system path to your sexy new executables.

Pretty soon your path can look like mine:

C:\Python26\Scripts;C:\Python26\;C:\Perl\site\bin;C:\Perl\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\Tools\sleuthkit-win32-3.1.3\sleuthkit-win32-3.1.3\bin;C:\Tools\UnxUtils\usr\local\wbin;C:\Program Files\GnuWin32\bin;C:\Tools\sleuthkit-win32-3.1.3\sleuthkit-win32-3.1.3\bin


And my $PATH is dead sexy.

Best. Job. Ever.

So, I promised exciting news last post and then it dawned on me that it's only exciting to my family and I. Sorry for the letdown.

I started my new job on Monday the 27th. Not just any old new job, but a job with Trustwave's Spiderlabs. If you don't know who who or what Spiderlabs are, start with the company webpage, then look at all the DEFCON talks, whitepapers, blogs and everything else these guys are responsible for.  They are the top Incident Response and Computer Forensics team in the country (my opinion of course) and I am absolutely stoked about joining them.

How on earth did I land a job with Trustwave?

For starters I have a lot of very relevant experience. I can walk into a server closet and discern what goes to what and why within a few minutes. This is important when every case situation is a total unknown.

I also went out of my way on my own dime to go to DEFCON and meet prospective employers. I shook a lot of hands, bought some drinks and asked some good questions. (see: networking)

I got certified earlier this year(see my earlier posts on studying for the GCFA) and started a forensics division at my former employer, business was just starting to build up when I got this offer.

Lastly, I may be the single luckiest person I know!  Timing, luck and a personal relationship with one of their senior consultants all came together at just the right time and led to job interviews. The rest is now history.


So what's a new guy learn on his first 2 cases?

1) People are still making the same simple mistakes when it comes to System Administration and auditing. Plain and simple. If you open RDP, PCAnywhere or VNC up to the internet and leave a weak, default or blank password, consider yourself pwned!

2) Hackers are getting better and better at disguising malware as valid processes. I can't go into a lot of detail here but plain sight is still the best hiding place of all.

3) P.O.S. integrators are screwing their customers! This is not to say that there are not some good integrators out there, but seriously, you cannot just drop these systems into place and pay absolutely no attention to the basic security fundamentals. When you do, you wind up costing your customers tens and possibly hundreds of thousands of dollars in investigations and fines.  Buck up! Put in a Netgear Prosafe for $85 and change those default passwords......or don't, I guess it's job security.


Thanks for following along.

Grayson