środa, 12 czerwca 2013

Limiting outgoing connections by max TPS

Interesting problem showed up. A customer requires that we never exceed some agreed number of TPS against his service (SOAP WS). On our side we've got couple of instances generating the traffic. They  in fact act as a pipe, processing the incoming requests. It seems a bit tricky to get all the instances to communicate with themselves extremely quickly to exchange the TPS information each of them has (sure, can be done - and I would use ØMQ for that if needed).
To sum up -  we have no influence on the incoming load, which comes from another customer (it's Google, actually ;)). Still, we have to stop all our requests to be forwarded. We need to start rejecting them above the threshold - queuing waiting for the less load would leave us with growing latency and with exhausted http connection pool in front of our app.
So, the idea is to use (already existing) reverse proxy  between the app and the outgoing customer's service. The xinetd's config comes in handy (from man page):

    cps

Limits the rate of incoming connections. Takes two arguments. The first argument is the number of connections per second to handle. If the rate of incoming connections is higher than this, the service will be temporarily disabled. The second argument is the number of seconds to wait before re-enabling the service after it has been disabled. The default for this setting is 50 incoming connections and the interval is 10 seconds.

The plan is to set it up to 'cps 29 0' (the goal is never exceed 30TPS) and see what happens :)

Different setting for the interval would result, in case of constant traffic like 31TPS, with loosing all 31 requests every 2nd second. Not good.

OK, let's see if the interval '0' works as expected... I'll update with results.


UPDATE:
finally, I have found  xinetd useless for the task. The thing is that (just took a look into the source code), when cps condition is reached, the service is stopped. And by this I mean that it is literally totally stopped... it rejects everything.  Actually, with that knowledge and reading the above spec now I see it states exactly that :) But still - it was surprising - I was rather expecting it to stop accepting more connections,  not to drop all undergoing work.

Well... the final solution is the most simple one: the TPS is monitored by the app itself. It is running on 2 separate instances, not aware about each other.  Both are set to work with max 14TPS. Of course it won't ever meet the max 30TPS, because it now all depends on the load balancer before the apps... Anyway, it is a conscious decision of the customer which claims that it will never reach so high rate anyway, so why should I care more than them? "Customer is King" :) - I did my job clearly explained all the consequences.

I have just used the com.google.common.util.concurrent.RateLimiter (from Google's Guava lib):

public synchronized static void checkIfBelowMaxTPS(final FRBYGConfig config, final String provider) throws MaxLimitProcessingException {

        RateLimiter rateLimiter = provider2RateLimiterMap.get(provider);

        //--- just for a double-check/logging;
        long now = System.currentTimeMillis(); //watch-out - different precision depending on OS (e.g. 10ms steps)
        requestsSet.add(now);
        Long oneSecBefore = now-1000-1; //-1 is because the subSet() is exclusive
        SortedSet<Long> subset = requestsSet.subSet(oneSecBefore, Long.MAX_VALUE);
        int noReqWithin1Sec = subset.size();
        requestsSet = subset;
        //---

        log.info("checking if the TPS is not above the treshold (provider="+provider+", limit="+rateLimiter.getRate()+"tps" + " - the current requests rate is: "+noReqWithin1Sec+"tps");
        if (!rateLimiter.tryAcquire()) {
            throw new MaxLimitProcessingException(...);
        }
    }


Also, just for a double check and logging purpose (RateLimiter doesn't provide runtime stats usable for logging) I added separate calculations there. It usually agree with RateLimiter, but is not 100% accurate. I have decided to trust RateLimiter in this task though.


piątek, 7 czerwca 2013

A new (ergonomic) keyboard

I decided to get myself a treat and to ease my aches caused by long hours of typing and get a new super-comfortable keyboard.

I looked up a possibilities first - it really seems there is just a couple of such things in the marked now, I was really surprised. One can find lots of gamer-dedicated mice, keyboards, special additional keyboards etc. But, there are just a few ergonomic keyboards meant for work. And - they are awfully expensive. My dream keyboard would be one of the Kinesis products e.g. this one:


But, the price - $300? WTF? Just for the simpler version...

So, surprisingly, I ended up with a Microsoft's keyboard - Natural Ergonomic 4000.




I have to say it is pretty nice, the price is OK as well (under 50Eur). I took the cable version over the wireless - batteries pisses me off and I work usually by the desk. The lining under wrists is a soft cushion, the special keys are working flawlessly under Linux - the Favorites, Home, Search, Back, Forward, Play, Volume controls. Really comfy, the angle the 2 pieces are set could be regulated though. I don't think I can manage to make the 'Zoom' switch to work, since xav doesn't show any keyboard input on them. Pity.
There are some flaws though - one is that the spacebar work not smoothly when pressed on the left/right edge (and it is meant to be used like this). The printings on the keys is not precise (like the paint got smudged). Overall - I like it, the paint I don't care since I do not look at the keyboard while typing, the spacebar seems to get working better with time. Now I have to try to use this pod in front edge of the keyboard - seems like a bit of revolution to the overall posture, but it would require sitting higher a bit (or have lower desk). I'll try.

The great thing I love is that the 'Microsoft' key is big and it is my mod key in Xmonad :)

Ecllipse 4.2 - slow as hell

I have just installed the brand new Eclipse 4.2 Juno. It is pretty (who cares!?) but - it is soooo slower than the previous 3.x version I had (Indigo).
Looked up what I can do - unfortunately this seems to be a result of a new UI feature - a user can now use CSS to modify the look&feel. It came with a price though.

I started up with modyfying the eclipse.ini the following way:

-startup
plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.200.v20120913-144807
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile
-vm
/opt1/soft/java/jdk1.7.0_13/bin/java
-vmargs
-Xmn128m
-Xms1024m
-Xmx1024m
-Xss2m
-XX:PermSize=128m
-XX:MaxPermSize=128m
-XX:+UseParallelGC


It significantly improved the start-up time and an overall responsiveness.
Still - it is terrible.

Then, I switched to Classic look&feel (Preferences->General->Appearance) and made sure animations are disabled. It has improved things - just a tad.

Anyway, GUI is still awfully slow (switching tabs, editing pom.xml) comparing to the previous version. 

Looked up in the net - seems there's nothing which can be done.
Just to make things clear - my machine is pretty fast:  i7, 8 cores, 8GB RAM, SSD.


The other surprise  - the Content Assists didn't work for Java code!
Seems it was just disabled by default - Preferences>Java>Editor>Content Assist>Advanced... WTF? 

Just downloaded Intellij Idea...