Discussion afterwards in #elasticsearch on Freenode:

11:04 <dakrone> mikepea: you have a very high segment count and it looks like the segments memory_in_bytes is about 1.2gb out of your 3gb heap
11:04 <dakrone> mikepea: also, it looks like you're using swap (which will make things very slow for JVM-based processes!)
11:05 <dakrone> mikepea: how many shards does this cluster have?
11:05 <mikepea> just the one
11:05 <mikepea> I'd say it's a typical 'small' ELK stack box. 
11:06 ⇐ johnraz_ quit (~johnraz@unaffiliated/johnraz) Ping timeout: 245 seconds
11:06 <dakrone> one shard? that is a lot of segments for a single shard
11:06 <dakrone> how big is the index?
11:06 <dakrone> size and document-wise
11:08 <mikepea> dakrone: 33GB for all indices, currently there are 44 indices (logstash daily pattern)
11:09 → costin_ joined ([email protected])
11:09 <mikepea> is indices->docs->count the doc count? In which case 33 million docs.
11:10  → jluis joined  ⇐ costin, falood and ArnaudM quit  
11:12 <dakrone> mikepea: oh okay, segment count is okay for 44 indices, in that case, since it's just lucene segment memory, that seems pretty reasonable, you have only the single node?
11:13 <mikepea> yeah, at the moment. Considering the potential of creating a larger stack, but for now it's not cost justifiable to have many ES nodes in the stack. 
11:13 ⇐ darkelda quit (~darkelda@unaffiliated/darkelda) Ping timeout: 265 seconds
11:15 <dakrone> okay, well if you can't add nodes, you could delete or close older indices, which would free the lucene memory associated with them
11:15 <mikepea> i guess this is a 'tuning ELK' question really. How can we reduce the segments memory_in_bytes count for an ELK system that's largely concerned with quick query of the current day's index.
11:15 <mikepea> is there any way of saying 'load this index into segment memory dynamically'?
11:16 <dakrone> mikepea: something you could do is optimize older indices that you are not indexing into into a single segment, see: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize
11:16 <dakrone> set max_num_segments to 1 in that case
11:16 ⇐ Xylakant quit ([email protected]) Quit: Leaving...
11:16 <dakrone> you can also use the curator tool to do it
11:18  → nickjj joined  ⇐ gschuring quit  
11:19 <mikepea> that sounds like a fine plan. We already have curator in place, but just for purging old data. I'll start graphing  segments memory_in_bytes and enable optimizing in curator.
11:19 <mikepea> thanks a million, this was gold knowledge. I'll update the gist with this discussion :)

As a follow-up, I configured curator to run daily as curator optimize --older-than 3 --max_num_segments 1

This dropped segments (on a test system) down from 600ish to 300ish, and heap seemed to reduce to 40% (from 80%). Interestingly segment memory (on the test system) only reduced from 250MB to 225MB.

I'll post results for the production system above when it has been applied.