Questions about netstat on Mac OS X

[Dave Taylor]

I'm working on some shell scripts for a new book I'm writing for O'Reilly &
Associates, and am stalled out on a script that analyzes netstat output.
It's a two part script: the first is run every 10 minutes from cron and logs
current net activity, then the second will ostensibly analyze the output to
highlight anything odd or curious happening. What is tripping me up is that
I really don't understand netstat output well enough and am hoping someone
on this list can illuminate it for me. I'm happy to share the resultant
script(s) back on the list if there's interest.

1. What are the different tcp devices shown?

$ netstat -d -I en0
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll Drop
en0 1500 <Link#5> 00:30:65:3d:e8:10 11538 638 9629 0
0 0
en0 1500 fe80:5::230 fe80:5::230:65ff: 11538 - 9629 -
- -
en0 1500 192.168.1 192.168.1.105 11538 - 9629 -
- -
en0 1500 (16)00:00:ff:be:76 11538 638 9629 0
0 0

I can see that there are input errors here, but on what devices? I assume
that only the device associated with my real IP is non-LAN traffic: am I
wrong?

2. What's the meaning of the various lines of output of ...

lines I think are important are highlighted in red

OSX $ netstat -s -p tcp
tcp:
26784 packets sent
20573 data packets (2822503 bytes)
7 data packets (4861 bytes) retransmitted
0 resends initiated by MTU discovery
3013 ack-only packets (1421 delayed)
0 URG only packets
0 window probe packets
1719 window update packets
1472 control packets
29101 packets received
12376 acks (for 2823103 bytes)
460 duplicate acks
0 acks for unsent data
26368 packets (8456463 bytes) received in-sequence
10 completely duplicate packets (3217 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
123 out-of-order packets (77469 bytes)
0 packets (0 bytes) of data after window
0 window probes
22 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
711 connection requests
72 connection accepts
1 bad connection attempt
0 listen queue overflows
770 connections established (including accepts)
975 connections closed (including 155 drops)
9 connections updated cached RTT on close
9 connections updated cached RTT variance on close
4 connections updated cached ssthresh on close
6 embryonic connections dropped
12362 segments updated rtt (of 12349 attempts)
23 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
14 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
653 correct ACK header predictions
15812 correct data packet header predictions

What I'm trying to do is to have the script show when you're seeing problems
(too many collisions, too many bad connection requests, etc) on your system,
but I'm, um, operating a bit in the dark. Let me ask this another way too,
perhaps: if you wanted to track your network performance, what would you
track, how would you do it, and how would you interpret the results? (for
example, 711 connection requests versus 72 connection accepts suggests that
there are a lot of incoming requests rejected by my firewall, yes?)

Thanks!!!

Dave Taylor

ps: I'm also looking for both some xferlog files that include anon ftp
transactions for analysis. Anyone have something they can share?

[Justin Walker]

[I sent this around noon, but haven't seen it show up; apologies if you
see multiple copies]

Hi, Dave,

I'll help where I can...

Post by Dave Taylor
I'm working on some shell scripts for a new book I'm writing for
O'Reilly &
Associates, and am stalled out on a script that analyzes netstat
output.
It's a two part script: the first is run every 10 minutes from cron
and logs
current net activity, then the second will ostensibly analyze the
output to
highlight anything odd or curious happening. What is tripping me up is
that
I really don't understand netstat output well enough and am hoping
someone
on this list can illuminate it for me. I'm happy to share the resultant
script(s) back on the list if there's interest.
1. What are the different tcp devices shown?

Do you mean "network devices"? The "-/-I" output will print a line for
each "sockaddr" that has been attached to the device's descriptor in

Post by Dave Taylor
$ netstat -d -I en0
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Coll Drop
en0 1500 <Link#5> 00:30:65:3d:e8:10 11538 638 9629
0 0 0

This is AF_LINK (i.e., the MAC[ethernet]) address.

Post by Dave Taylor
en0 1500 fe80:5::230 fe80:5::230:65ff: 11538 - 9629
- - -

This is AF_INET6 (IPv6) - the dreaded 128-bit address, written as
16-bit hex chunks, with leading zeros elided, and "::" permitted once
to indicate a long string of zeros.

Post by Dave Taylor
en0 1500 192.168.1 192.168.1.105 11538 - 9629
- - -

This is AF_INET (IPv4) - the usual "dotted-quad" IP address.

Post by Dave Taylor
en0 1500 (16)00:00:ff:be:76 11538 638 9629
0 0 0

This is the AppleTalk address. The parenthetical '16' is the value of
AF_APPLETALK (given in /usr/include/sys/socket.h); 'netstat' prints the
AF_ value in parens, and the address as a hex string, if it doesn't
recognize the family (i.e., no-one bothered to educate 'netstat' about
AppleTalk).

Post by Dave Taylor
I can see that there are input errors here, but on what devices? I
assume
that only the device associated with my real IP is non-LAN traffic: am
I
wrong?

You'll notice that the repeated values are always the same. This is
because each line refers to the same device. In some cases, some
values aren't printed (e.g., for the AF_INET line, the ierrs value
doesn't show up). This is because the value in question may not
directly relate to the AF entity [here, the ierrs count input errors of
some kind but those don't make sense for a higher-layer protocol like
IPv4; the device can't generally account for errors based on packet
type - if there's an error, the content is suspect; for AppleTalk, the
AF_ is unknown to 'netstat', so it goes ahead and displays the value].

Post by Dave Taylor
2. What's the meaning of the various lines of output of ...

I get this in plain ASCII (cf. my signature), so this doesn't come
through.

Post by Dave Taylor
OSX $ netstat -s -p tcp

All of the lines are important. For example, a lot of bad checksum or
other discard reasons would indicate a serious problem - whether it's
hardware or some kind of DoS attack is in questoin. Some of the lines
are just "statistics", while others are error counters. Generally, you
can tell by the name. For diagnosing real problems, all may play a
role, but without a real problem, you can't predict which ones will.

Post by Dave Taylor
What I'm trying to do is to have the script show when you're seeing
problems
(too many collisions, too many bad connection requests, etc) on your
system,
but I'm, um, operating a bit in the dark. Let me ask this another way
too,
perhaps: if you wanted to track your network performance, what would
you
track, how would you do it, and how would you interpret the results?

This is not something you can do in one chapter of a book. This is
what's called ART, and it requires a lot of experience. One of the
best ways to do this is to look at lots of examples: e.g., take a
sample every hour, and review them daily. When you actually get
reports of a performance problem, try to narrow down the occurrence and
relate it to your stats.

Post by Dave Taylor
(for
example, 711 connection requests versus 72 connection accepts suggests
that
there are a lot of incoming requests rejected by my firewall, yes?)

Hard to tell. If by "incoming request" you mean the TCP packet with
the SYN bit set, then if such were rejected by the firewall, you
wouldn't see it as a request (the firewall gets to see the packet
before the TCP engine). If the firewall can reject a request after the
SYN packet has been passed, then this could be an explanation.

Hope that helps. Ask again if I've misunderstood or gone awry in the
above.

Regards,

Justin

--
/~\ The ASCII Justin C. Walker, Curmudgeon-at-Large
\ / Ribbon Campaign
X Help cure HTML Email
/ \


--
Justin C. Walker, Curmudgeon-At-Large *
Institute for General Semantics | If you're not confused,
| You're not paying attention
*--------------------------------------*-------------------------------*

[Justin Walker]

Hi, Dave,

I'll help where I can...

Post by Dave Taylor
I'm working on some shell scripts for a new book I'm writing for
O'Reilly &
Associates, and am stalled out on a script that analyzes netstat
output.
It's a two part script: the first is run every 10 minutes from cron
and logs
current net activity, then the second will ostensibly analyze the
output to
highlight anything odd or curious happening. What is tripping me up is
that
I really don't understand netstat output well enough and am hoping
someone
on this list can illuminate it for me. I'm happy to share the resultant
script(s) back on the list if there's interest.
1. What are the different tcp devices shown?

Do you mean "network devices"? The "-/-I" output will print a line for
each "sockaddr" that has been attached to the device's descriptor in

Post by Dave Taylor
$ netstat -d -I en0
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Coll Drop
en0 1500 <Link#5> 00:30:65:3d:e8:10 11538 638 9629
0 0 0

This is AF_LINK (i.e., the MAC[ethernet]) address.

Post by Dave Taylor
en0 1500 fe80:5::230 fe80:5::230:65ff: 11538 - 9629
- - -

This is AF_INET6 (IPv6) - the dreaded 128-bit address, written as
16-bit hex chunks, with leading zeros elided, and "::" permitted once
to indicate a long string of zeros.

Post by Dave Taylor
en0 1500 192.168.1 192.168.1.105 11538 - 9629
- - -

This is AF_INET (IPv4) - the usual "dotted-quad" IP address.

Post by Dave Taylor
en0 1500 (16)00:00:ff:be:76 11538 638 9629
0 0 0

This is the AppleTalk address. The parenthetical '16' is the value of
AF_APPLETALK (given in /usr/include/sys/socket.h); 'netstat' prints the
AF_ value in parens, and the address as a hex string, if it doesn't
recognize the family (i.e., no-one bothered to educate 'netstat' about
AppleTalk).

Post by Dave Taylor
I can see that there are input errors here, but on what devices? I
assume
that only the device associated with my real IP is non-LAN traffic: am
I
wrong?

You'll notice that the repeated values are always the same. This is
because each line refers to the same device. In some cases, some
values aren't printed (e.g., for the AF_INET line, the ierrs value
doesn't show up). This is because the value in question may not
directly relate to the AF entity [here, the ierrs count input errors of
some kind but those don't make sense for a higher-layer protocol like
IPv4; the device can't generally account for errors based on packet
type - if there's an error, the content is suspect; for AppleTalk, the
AF_ is unknown to 'netstat', so it goes ahead and displays the value].

Post by Dave Taylor
2. What's the meaning of the various lines of output of ...

I get this in plain ASCII (cf. my signature), so this doesn't come
through.

Post by Dave Taylor
OSX $ netstat -s -p tcp

All of the lines are important. For example, a lot of bad checksum or
other discard reasons would indicate a serious problem - whether it's
hardware or some kind of DoS attack is in questoin. Some of the lines
are just "statistics", while others are error counters. Generally, you
can tell by the name. For diagnosing real problems, all may play a
role, but without a real problem, you can't predict which ones will.

Post by Dave Taylor
What I'm trying to do is to have the script show when you're seeing
problems
(too many collisions, too many bad connection requests, etc) on your
system,
but I'm, um, operating a bit in the dark. Let me ask this another way
too,
perhaps: if you wanted to track your network performance, what would
you
track, how would you do it, and how would you interpret the results?

This is not something you can do in one chapter of a book. This is
what's called ART, and it requires a lot of experience. One of the
best ways to do this is to look at lots of examples: e.g., take a
sample every hour, and review them daily. When you actually get
reports of a performance problem, try to narrow down the occurrence and
relate it to your stats.

Post by Dave Taylor
(for
example, 711 connection requests versus 72 connection accepts suggests
that
there are a lot of incoming requests rejected by my firewall, yes?)

Hard to tell. If by "incoming request" you mean the TCP packet with
the SYN bit set, then if such were rejected by the firewall, you
wouldn't see it as a request (the firewall gets to see the packet
before the TCP engine). If the firewall can reject a request after the
SYN packet has been passed, then this could be an explanation.

Hope that helps. Ask again if I've misunderstood or gone awry in the
above.

Regards,

Justin

--
/~\ The ASCII Justin C. Walker, Curmudgeon-at-Large
\ / Ribbon Campaign
X Help cure HTML Email
/ \

[William H. Magill]

I'm not a netstat expert, but here's a couple of data points

Post by Dave Taylor
1. What are the different tcp devices shown?
$ netstat -d -I en0
Name  Mtu   Network       Address            Ipkts Ierrs    Opkts
Oerrs  Coll Drop
en0   1500  <Link#5>    00:30:65:3d:e8:10    11538   638     9629
    0     0   0
en0   1500  fe80:5::230 fe80:5::230:65ff:    11538     -     9629
    -     -   -
en0   1500  192.168.1     192.168.1.105      11538     -     9629
    -     -   -
en0   1500  (16)00:00:ff:be:76               11538   638     9629
    0     0   0
I can see that there are input errors here, but on what devices?  I
assume that only the device associated with my real IP is non-LAN
traffic: am I wrong?

There is only one device here - en0, the Ethernet interface known as
zero. That interface has both an IPv4 and IPv6 stack associated with it.

lo0 is the loop-back interface.
en1 is (usually) the Airport interface, but might just be a second card.

This set of report descriptions comes from the Tru64 Unix man page:
The network interface display format provides a table of cumulative
statis-
tics for the following:
+ Interface name
+ Maximum Transmission Unit (MTU)
+ Network Address
+ Packets received (Ipkts)
+ Packets received in error (Ierrs)
+ Packets transferred (Opkts)
+ Outgoing packets in error (Oerrs)
+ Collisions
Note that the collisions item has different meanings for
different
network interfaces.
+ Drops (optional with -d)
+ Timers (optional with -t)

Note that the -d option is listed/defined in the OS X man page, but the
-t option is only listed. (Man pages are not known for their accuracy
or completeness.)

Post by Dave Taylor
2. What's the meaning of the various lines of output of ...
lines I think are important are highlighted in red
OSX $ netstat -s -p tcp

Don't mix apples and oranges. This command generates TCP stats across
ALL interfaces.

This might map to en0 if that is the ONLY port with TCP/IP traffic, but
not necessarily. A box with both an Airport and an Ethernet for
instance, if the laptop runs hardwired some times, and wireless at
others.

As I recall, the counters are not zeroed except on boot.

I assume that you realize the counters are cumulative and you will need
to subtract them to get the "interval count." I believe that they are
also 32 bit counters, but they might be less. Somebody will have to
look at the code to find out.

Netstat, like ping and traceroute are very low level network debugging
tools. They are useful for trouble shooting, not tracking performance.
They are oriented at hardware issues, not traffic. While you can infer
performance issues with some of their statistics, that's not what they
were designed to convey. (Note that I'm talking about the Interface
options of netstat here.)

On a properly configured and active (more than one host) ethernet
SEGMENT, there will always be collisions. Note that a SEGMENT is either
a HUB or single cable with multiple Ethernet devices connected to it. A
switched environment, however, should always have zero collisions. This
is true for some hub configurations as well. (Maybe, most hubs today.)
Similarly, a WiFi (802.11) connection with only one station active will
not likely see collisions, but with multiple stations transmitting, you
will begin to see collisions. A WiFi Access point is basically a Hub.

Some problems -- like duplex-mismatch -- occurring on a switch won't
show up at all in netstat, but your performance will be going to hell
in a handbasket. This is because the problem is happening in the media
layer below netstat's view.

Off hand, I suspect that you'll have to dig up a copy of Comer (TCP/IP
Networking) to find out what all of the Kernel data structures "really"
mean (the -s output). I don't recall seeing them documented anywhere.
It's always assumed that if you care, you know what they mean
otherwise, they have labels which are "self expalanatory." Sometimes
SNMP documentation (which doesn't exist for OS X that I know of)
contains useful information as it is another way of displaying these
same data structures.

And by the by, you probably should be doing this with Perl rather than
a shell script. For that matter, there may already be some such critter
floating around.

T.T.F.N.
William H. Magill
# Beige G3 - Rev A motherboard - 768 Meg
# Flat-panel iMac (2.1) 800MHz - Super Drive - 768 Meg
# PWS433a [Alpha 21164 Rev 7.2 (EV56)- 64 Meg]- Tru64 5.1a
***@mcgillsociety.org
***@acm.org
***@mac.com