Security Things
Validate Backend Servers
If you have secure data, and doing SSL to back end, should ensure netscaler checks validity of certs on services. By default it does not, which means basically its just doing ip address based authentication.
set ssl service
Updating SSL keys:
Make sure you use:
Update ssl certkey
to update SSL certs – otherwise you need to unbind, remove the old certkey (as two identical certificates with the same “Subject-Identifier” and “Issuer-Identifier” cannot be loaded in the kernel), add new cert and bind again – this means a few seconds of downtime.
Header Insertions
If doing header insertion (for client IP, etc) should drop requests coming in that have that header. Netscaler will just add additional header if it exists, which could lead to insecure or indeterminate behaviour in app if it depends on header.
add service www1 -http www1 HTTP 80 -gslb NONE -maxClient 125 -maxReq 10000 -cacheable NO -cip ENABLED ClientHost
add policy expression ClientHostHead HTTPHEADER ClientHost EXISTS
add ns filter NoClientHost -reqRule ClientHostHead -reqAction RESET
Debugging things
What events did the netscaler see? Services passing/failing healthchecks? Very useful.
nsconmsg -K /var/nslog/newnslog -d event
2246 0 ’server_NSSVC_HTTP_216.52.45.145:80(test)’ UP Thu Jul 26 00:43:00 2007
2255 0 ’server_NSSVC_HTTP_216.52.45.174:80(test-vip)’ UP Thu Jul 26 00:44:29 2007
2257 58522 ’server_NSSVC_HTTP_216.52.45.145:80(test)’ Out Of Service Thu Jul 26 00:45:28 2007
2258 0 ’server_NSSVC_HTTP_216.52.45.174:80(test-vip)’ DOWN Thu Jul 26 00:45:28 2007
Was the netscaler sending traffic to various services?
nsconmsg -K /var/nslog/newnslog -s ConLb=1 -d oldconmsg | grep “time\|IP OF SERVICE or VIP”
See how things are doing:
nsconmsg –d oldconsmsg –s FIELD
Case sensitive for Field.
nsdebug_pe 1 = interface debug
ConDebug Connection info debug. 1= basic, 2= detailed, 3= all sorts of stuff about internal TCP parameters
ConLb 1= Load balancing debug
ConCSW 1=content switching debug
ConSSL 1=ssl Debug
ConCMP 1=compression debug
ConIC 1=integrated caching debug
e.g. Evaluate compression:
nsconmsg -s ConCMP=1 -d oldconmsg
CMPResps:CRes=547 Cin=20304690 Cout=6830730 Cratio=2.97(34%)
Response: Res=17649 Rin=161486642 Rout=148012682 Rratio=1.09(92%)
Compressible traffic being compressed by 66%; total only 8%
nsconmsg -s ConDebug=1 -d oldconmsg
Displaying debug performance information
Performance Data Record Version 2.0
current time is Thu Jul 19 11:56:23 2007
HTTP: Req(41580876512 1.1(39141733520) 1.0(1733429699)Get(38133042089) Postp(1966228573) Others(1481605850)) Res(41496471614 1.1(40630248963) 1.0(866222651) Pipe(11644297))
HTTP: Req/s(2623 1.1(92%) 1.0(5%) time=1) avgReq/s(0 1.1(0%) 1.0(0%) time=0)
HTTP: Res/s(2602 1.1(95%) 1.0(4%) time=1) avgRes/s(0 1.1(0%) 1.0(0%) time=0)
Note: 5% of requests are HTTP1.0. Oddly, so are 4% of responses. (Old servers?)
Examine response time (Time to first byte) of services, vservers:
To see live data:
nsconmsg -f “*svr_ttfb*” -d current
To see data in current log file, from start of log file:
nsconmsg -K /var/nslog/newnslog -f “*svr_ttfb*” -d current #historical
Nstcptrace.sh – very handy.
Can also use
/etc/nsapimgr -K nstrace3 -d netraces
to look at trace files saved with nstrace
Please send me any Netscaler chat groups available.
I am mostly interested in axploiting all dashboared & logging / debugging features of Netscaler 8.0 – still have a couple of 6.1
Best dashboarding system I have seen for Netscalers by far is http://www.logicmonitor.com/ – automatically discovers all VIPs, monitors traffic and requests for each, as well as full break down by content switching servers, integrated caching, compression, etc.
Check it out – it makes monitoring Netscalers trivial, instead of a total pain.
Thank you very much for the logmonitor & the rest of the Netscaler support provided here.
Highly appreciated! Straight forward commands…
When the primary fails & it is not reachable, what type of commands are available after a reboot to check what went wrong other than it went down?
I am using three independent clusters of 9000 NS.
Thanks,
Ron Ikonomou
If the netscaler crashes, there should be some indications of the failure once you drop to the shell, in:
/var/log/messages
kernel dumps in:
/var/crash
Of course, most of the reasons will not mean much except to Netscaler support.
e.g.
/flash/ns-8.0-55.3: kernel trap 28 with interrupts disabled at eip=305f6810
ns /flash/ns-8.0-55.3: Fatal trap 28: machine check trap while in kernel mode
So the best recourse is usually to open a case, or just upgrade to most recent GD release (as that is usually Netscaler support’s recommendation.)
If you think you have a new bug, then open a case, certainly.