Full Content Data

From NSMWiki

The following are example Full Content (think Snort as packet logger via log_packets.sh) disk usage scenarios for various production sensors. Please follow the initial templates when adding your information. Thank you!

Data Collection Methodology

Disk Usage: This is the amount of space occupied by the snort.log.$TIMESTAMP files in /nsm/$SENSOR/dailylogs for the period in question (30 Jun - 13 July, inclusive).

$ du -csh 2007-06-30 2007-07-0* 2007-07-10 2007-07-11 2007-07-12 2007-07-13

• Example 1: Sguil installation
  • Period Collected: 14 days
  • Monitored Link: 6 Mbps (four bonded T-1s)
  • Maximum Bandwidth: Unknown
  • Average Bandwidth: Unknown
  • Disk Usage: 75 GB

2.7G    2007-06-30
2.6G    2007-07-01
6.1G    2007-07-02
6.8G    2007-07-03
2.7G    2007-07-04
6.9G    2007-07-05
7.3G    2007-07-06
3.3G    2007-07-07
3.0G    2007-07-08
6.9G    2007-07-09
6.6G    2007-07-10
6.8G    2007-07-11
6.4G    2007-07-12
6.4G    2007-07-13
 75G    total

• Example 2: Sguil installation
  • Period Collected: 14 days
  • Monitored Link: 6 Mbps (four bonded T-1s)
  • Maximum Bandwidth: Unknown
  • Average Bandwidth: Unknown
  • Disk Usage: 46 GB

856M    2007-06-30
723M    2007-07-01
4.3G    2007-07-02
4.1G    2007-07-03
746M    2007-07-04
5.9G    2007-07-05
3.4G    2007-07-06
716M    2007-07-07
586M    2007-07-08
5.4G    2007-07-09
4.5G    2007-07-10
6.6G    2007-07-11
3.7G    2007-07-12
4.8G    2007-07-13
 46G    total

Disk Usage: This is the amount of space occupied by the snort.log.$TIMESTAMP files in /nsm/$SENSOR/dailylogs for the period in question (13 July - 21 July, inclusive) using FreeBSD netgraph by connecting two nodes(NICs).

$ du -csh *

• Example 3: Sguil installation
  • Period Collected: 8 days
  • Monitored Link: 100 Mbps
  • Maximum Bandwidth: 75Mbps
  • Average Bandwidth: 10Mbps
  • Disk Usage: 1200 GB

 11G    2007-07-13
 32G    2007-07-15
102G    2007-07-16
116G    2007-07-17
198G    2007-07-18
334G    2007-07-19
325G    2007-07-20
 86G    2007-07-21
1.2T    total

For his own purposes, one NSM practitioner uses the rule of thumb that 1500 MB per 1 Mbps of traffic per day is needed for full content data. As an example, a 50% average utilization 100 Mbps link requires 75000 MB (75 GB) of disk space per day, so recording 14 days of session data requires 1333 GB (over 1 TB).