Enhanced logging with the Citrix NetScaler

There’s nothing more frustrating than logs that don’t actually log useful information (OK, maybe there are a few things that are more annoying).  But by and large, when you’re having a problem with one of your load balanced apps, you want to KNOW right now!

I have recently found the custom message action option, and have found it to be an incredibly useful tool.  Like everything else in NetScaler world, you bind the message action to some entity, a content switching virtual server policy for example, and off you go.

The message action enables you to pull specific information about the request or response that meets the criteria you set, and boom, it’s in your syslog traffic.  You can snag pretty much anything you want, URL, headers, POST data, whatever.    Pull it up in $loganalyzer and boom, instant analytics.  Here’s how you get started:

1. Set the basics

This tutorial assumes you are already set up and successfully syslogging to a remote host, but you can log this same data to the newnslog file too (although I don’t think there is a more painful way to try to look at log data than that, but whatever!).

set audit syslogParams -userDefinedAuditlog YES

2. Define what your super awesome custom message will look like:

We ran into an issue where we had noticed the default policy on a content switched virtual server was taking lots of hots, and we wanted to see what was getting there.  We decided we’d need to know:

  • The IP address of the request
  • The HTTP method
  • The Host (from the header)
  • The URL (excluding all that crazy query stuff… ah .NET!)

We also decided that the message should make sense to a human reading it, not just to some search time extractions in your elastic search cluster….. So here’s our policy!

"Content switch policy hit for load-balanced-vserver:"+HTTP.REQ.LB_VSERVER.NAME+" ClientIP: "+CLIENT.IP.SRC+" issued a "+HTTP.REQ.METHOD+" request for "+HTTP.REQ.HEADER("Host")+HTTP.REQ.URL.HTTP_URL_SAFE

Let’s break this down:

  • Anything in quotes goes as a literal string in your syslog message (that’s how we make sure humans can read it), and you stich everything together with the plus (+)
  • HTTP.REQ.LB_VSERVER.NAME – returns the name of the load balanced virtual server handling the traffic.  We decided after we built the policy that if we made it correctly we could use it all over the place (we have TONS of content switching virtual servers)
  • CLIENT.IP.SRC – returns the IP making the request
  • HTTP.REQ.METHOD – returns the GET or POST or whatever method
  • HTTP.REQ.HEADER(“Host”) – the DNS name the user is using to access the service
  • HTTP.REQ.URL.HTTP_URL_SAFE.PATH – the URL that was requested, properly sanitized (watch out for second order XSS!), with just the path (it throws away anything from the ?<query>=)

We decided to use just the path and not capture the entire URL because adding the additional query characters didn’t matter (we almost never make content switching decisions based on that), it made the logs harder to read, and when we tried to analyze the results they just muddied the waters by making every request look unique.

A couple of things to watch out for here:

Don’t mix up request and response header stuff, it doesn’t work like that (at least not that we can find).

If you bypass the safety check, make sure you don’t expose yourself to risk later.

Hope this helps add some insight into your load balancing operation!

Leave a Reply