View source

One thing that would be really helpful to me would be the ability to extract ONLY the http headers (and even only the first few lines of the headers).
Ideally, I'd like to be able to see this for not just one 'session', but for an entire sancp query result... e.g. if I do a sancp query srcIP/1hour, I'd like to see a transcript of, or export to wireshark, the ENTIRE query result.
Can that be done (is there a way to do it in sguil already)?
i.e.
Right now, I can click on an event, and view the 'transcript', but if
there's images, large html, ..., it's hard to get a quick read for what
web pages were requested and what the response was.
Is there a way to have the transcript only display the first X bytes from
each packet?
I do see that I can type 'http' into wireshark and basically get what I
want, but only for one transcript at a time.
Other methods I can think of for doing this, besides viewing the
transcript directly or using wireshark, are:
a. run the tcpdump-formatted transcripts through something that just
extracts http requests and responses.
DSniff's URLSnarf is one such program.
I recently saw another program like urlsnarf that also printed the server
response code, but I've forgotten the name of it.
Would it be easy to add this filtering functionality to the packet agent?
Could this be done for an entire sancp or other query resultset?
b. have a sguil agent on the web server that could receive a request from
sguild (like how the packet agent works), parse the IIS or CLF logs, and
return all results from the IP in whatever timeframe
disadvantage: very slow if logs are big
c. download all the transcripts and run them through a perl script. I have
an ngrep/perl script already that could do what I want, although it
doesn't show the server response code.
This only would have a significant advantage over using wireshark IF
there's a way to get ALL of the transcripts from a sancp query in one
operation; preferably saved to their own folder.
The output I'd like to see would look something like this:
the request, e.g. SRCIP DSTIP GET /url...
the server response, e.g. 404 not found, ...
e.g. URLSnarf's output:
 # urlsnarf -i eth1
 urlsnarf: listening on eth1 [tcp port 80 or port 8080 or port 3128]
 192.168.1.3 - - [23/Jul/2008:15:41:52 -0700] "GET http://suggestqueries.google.com/complete/search?output=firefox&client=firefox&hl=en-US&q=sguil HTTP/1.1" - - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1"
 192.168.1.3 - - [23/Jul/2008:15:41:52 -0700] "GET http://www.google.com/search?q=sguil&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a HTTP/1.1" - - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1"
Ideally, I'd like to only see requests for dynamic web pages (e.g. .php,
.aspx, ...), and filter out images, etc.
My ngrep/perl script handles that, but again, it doesn't show the response
code from the server.
After I posted the above to this list, Richard Bejtlich responded:
<blockquote>I plan to run a tool like this
http://taosecurity.blogspot.com/2008/06/logging-web-traffic-with-httpry.html
in parallel with my other NSM tools on my sensors.  It might be cool
to have those records available to Sguil.  I haven't thought that far
down the road yet.</blockquote>
That seems like it would meet my needs!.
--[[User:Barrygould|Barrygould]] 05:58, 29 July 2026 (UTC)