Extracting HTTP Request Headers
One thing that would be really helpful to me would be the ability to extract ONLY the http headers (and even only the first few lines of the headers).
Ideally, I'd like to be able to see this for not just one 'session', but for an entire sancp query result... e.g. if I do a sancp query srcIP/1hour, I'd like to see a transcript of, or export to wireshark, the ENTIRE query result. Can that be done (is there a way to do it in sguil already)?
i.e. Right now, I can click on an event, and view the 'transcript', but if there's images, large html, ..., it's hard to get a quick read for what web pages were requested and what the response was.
Is there a way to have the transcript only display the first X bytes from each packet?
I do see that I can type 'http' into wireshark and basically get what I want, but only for one transcript at a time.
Other methods I can think of for doing this, besides viewing the transcript directly or using wireshark, are:
a. run the tcpdump-formatted transcripts through something that just extracts http requests and responses. DSniff's URLSnarf is one such program. I recently saw another program like urlsnarf that also printed the server response code, but I've forgotten the name of it. Would it be easy to add this filtering functionality to the packet agent? Could this be done for an entire sancp or other query resultset?
b. have a sguil agent on the web server that could receive a request from sguild (like how the packet agent works), parse the IIS or CLF logs, and return all results from the IP in whatever timeframe disadvantage: very slow if logs are big
c. download all the transcripts and run them through a perl script. I have an ngrep/perl script already that could do what I want, although it doesn't show the server response code. This only would have a significant advantage over using wireshark IF there's a way to get ALL of the transcripts from a sancp query in one operation; preferably saved to their own folder.
The output I'd like to see would look something like this: the request, e.g. SRCIP DSTIP GET /url... the server response, e.g. 404 not found, ...
e.g. URLSnarf's output:
# urlsnarf -i eth1 urlsnarf: listening on eth1 [tcp port 80 or port 8080 or port 3128] 192.168.1.3 - - [23/Jul/2008:15:41:52 -0700] "GET http://suggestqueries.google.com/complete/search?output=firefox&client=firefox&hl=en-US&q=sguil HTTP/1.1" - - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:184.108.40.206) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1"
192.168.1.3 - - [23/Jul/2008:15:41:52 -0700] "GET http://www.google.com/search?q=sguil&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a HTTP/1.1" - - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:220.127.116.11) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1"
Ideally, I'd like to only see requests for dynamic web pages (e.g. .php, .aspx, ...), and filter out images, etc. My ngrep/perl script handles that, but again, it doesn't show the response code from the server.
After I posted the above to this list, Richard Bejtlich responded:
I plan to run a tool like this
in parallel with my other NSM tools on my sensors. It might be cool to have those records available to Sguil. I haven't thought that fardown the road yet.
That seems like it would meet my needs!.
--Barrygould 05:58, 29 July 2008 (UTC)