A case in point – a server had an alert that:

The host HOST is experiencing an unusual number of failed TCP connections.
There are now 2.05 % failed connections, putting the host in a warn level.
This started at  2011-01-13 07:23:02 PST.
This could be caused by incorrect application backlog parameters,
or by incorrect OS TCP listen queue settings.

So, back to handy dandy command line tools.
One seconds worth of tcpdump of traffic was captured.
I then ran tcptrace against the captured tcpdump file, using tcptrace’s excellent filtering:
tcptrace -f’syn_count=1&& segs<10′ /tmp/dump

I figured that if there were connections transitioning from SYN-RCVD state to CLOSED, then they’d have a SYN, and something odd in the next few packets, but there had to be very few packets in the connection.
So I told tcptrace to show me all connections that had a syn (in either direction) and less than 10 segments.

Turns out there were several such connections – all from the same host to the server, and all consisting of:
syn (from client), SYN-ACK (from server), and RESET from client, within milliseconds.
No loss, retransmits, etc – so all the issues are from this one client, but it is not a network issue. It’s application level, involving the app on that one client.

So we sent off word to the administrator of that client, he updates it’s software, issue resolved. (I don’t know exactly what the old software was doing, but it was clearly not working.)

So while monitoring is necessary and essential and alerted us to the problem, to figure out what the problem was often still requires other tools.
tcptrace is an excellent tool for such cases.

I double checked that /etc/snmpdv3.conf was allowing connections with community public from localhost – it was.

Higher log levels on aixmibd didn’t show anything more:
../../../../../../src/tcpip/usr/sbin/aixmibd/aix_main.C (1059) hostname=loopback, community=public
../../../../../../src/tcpip/usr/sbin/aixmibd/aix_main.C (1067) rc=-9 from DPIconnect_to_agent_SHM()
Fri Oct 15 10:10:15 CDT 2010 Connect to SNMP agent on the host loopback failed, will keep trying

But…it was actually connecting to a hostname called loopback, not the loopback address.
A quick ping and nslookup showed that for whatever reason, this customer had a host called “loopback” in DNS – which was resolving to a real system, not the loopback address. Hence the failures.
So, given that the /etc/hosts file already has an entry for loopback, just had to make it get used:
Appended
hosts = local 4, bind4
to
/etc/netsvc.conf
and all was well.

The AIX snmp agents all started correctly; we set up the LogicMonitor recommended config for net-snmp to proxy some requests to the AIX snmpd and answer the rest itself, and all the data, graphs and alerting started working as they should.

I dont think analysis of SQL is monitoring…

Because the Gear 6 appliance runs memcached on multiple IP addresses on the one system, instead of the multiple ports that is more common, in order to monitor a gear 6 appliance you need to enumerate all the IP addresses on the system, and monitor them via the memcached API. No problem, the memcached monitoring of LogicMonitor does that, all nice and automatically as you’d expect.
The issue is that each memcached instance on gear 6 has some purpose, which you can use as the name in the gear 6 config, but I wanted to see them in monitoring. Talking to LogicMonitor (as I am lazy, and wanted them to fix the issue as they usually do without work on my part), they said “We’d love to, but there is no way to get that info from the gear 6 box.”

Being the distrustful type, and recalling that in a quick look at the gear 6 MIB, I had seen the application name in the MIB tree, I investigated myself, but they were right. Gear 6 exposes the application name, and the IPs (through the standard address MIB), but doesn’t index the one to the other, so there is no way to associate them. Whatup Gear 6 people? Use the IP address as an index, or have it returned as a value in the globalService table.
Otherwise I have to go in and manually add in a comment to each discovered memcached instance, so I know what it is.

Then the VMware client will know about the hardware status.
But even then, it seems that the hardware monitoring is exposed differently – via CIM rather than the VMware API.
Never keep it simple when it can be complicated!

It means that the nice automatic overview graphs we’ve come to expect:
dont exist for NFS datastores.
We can get the same things from the individual VM’s, and creating a dashboard graph that displays the same data, but it seems this is something ESX should support.
(I know, an NFS call is not the same as a disk operation, but I’m just ranting.)

Looking at the various MySQL monitoring graphs below, you can see the “Hmm.” The release was on 8/5.
CPU Load dropped:

Query Cache had less queries hitting it:

All the graphed MySQL operations dropped:

But..
Total Mysql Questions went up…

OK, so we weren’t terribly worried – all the important metrics went the right way. There were less Innodb cache misses; so less disk IO, better query response time, etc.
But what was with the increase in total Questions per second? How could that even be when all the operations graphed went down?
That was exactly the right question to ask – given all the counters that were graphed decreased, it must be one that was not graphed by the monitoring system.
A quick ‘show global status’ showed the only counter with a big value that was not graphed was com_set_options.

The new application code sets options on every transaction – and the monitoring system didn’t (by default) graph com_set_options as it’s not usually a statement that affects performance. We added that counter to the data collection of the Mysql monitoring for this client, so we’ll be able to see it clearly when we update the code to not set so many options – and it just makes diagnosis easier. The more you know, the more you know.

Second hiccup – I now had a working NetApp running on my Linux host, but it could not communicate with any other host. I could see it’s ARP requests reaching the default gateway, and the gateway replying, by the replies were not being seen on the Linux host hosting the simulator, nor on the simulator itself.

This tickled the memory that VMware locks down the virtual switch from promiscuous mode (and the simulator puts the host Linux into promiscuous mode, so it can receive packets for the virtual NetApp, which has its own mac and IP).
so:
fire up vCenter, click the Host, Configuration, Networking, properties, and set the VM Network switch to accept Promiscuous mode, Mac address changes, and Forged transmits.

Now my virtual NetApp is reachable, I can mount its volumes from other hosts in the lab, and all is well.
It responds to NetApp monitoring software just like a real NetApp, too, with API monitoring, etc.