Wonderful Performance Metrics Tool

When my crawler project is closed to the end. my concern on the performance is heavier. How to measure my system performance? there is no easy answer for it. So many module, so many parameters, most important is all these CAN'T affect the system performance and increase the module complexity.  

Even for the JSON parsing function, I spend almost 1 working day to come out the performance measurement and it is just for the unit testing ;( . The result is just like the following. 

stockprice (36648000 records) elapsed ms:376806.531 for 36000 avg:9.937 variance:5.652 Fastest:9.000 Slowest:244.000

[67, 9 x 16910, 10 x 13029, 11 x 4306, 12 x 868, 73, 13 x 248, 14 x 123, 15 x 49, 17 x 15, 16 x 33, 19 x 9, 18 x 8, 21 x 26, 20 x 11, 23 x 51, 22 x 47, 25 x 43, 24 x 41, 27 x 39, 26 x 38, 29 x 24, 28 x 28, 31 x 3, 30 x 18, 34, 35 x 2, 32 x 2, 33 x 6, 38, 39 x 2, 36 x 6, 37 x 3, 42 x 2, 43, 41, 50, 48, 54, 244]

But how about other functions.... I was almost frightened by the future workload. It seems my system launch day need be postpone. Until today, I find this wonderful library, Metrics, through the netty example. It is fantastic and save me huge time on the performance measurement and reporting. 

With just few lines, the following result will be automatically printed into the System console. If you need, it can easily output the result into CSV, log file, JMX, even provide the servlet to remotely pass the result as JSON. Wonderful!!! 

With all these tools, I almost got the insurance on my system quality.

 final ConsoleReporter reporter = ConsoleReporter.forRegistry(Metrics.defaultRegistry())

                                                    .convertRatesTo(TimeUnit.SECONDS)

                                                    .convertDurationsTo(TimeUnit.MILLISECONDS)

                                                    .build();

reporter.start(1, TimeUnit.MINUTES);

Timer timer = Metrics.newTimer(this.getClass(),"StockPrice Batch Parse","timer",new SlidingWindowReservoir(nMax));

timer.update(stopwatch2.stop().elapsed(TimeUnit.NANOSECONDS),TimeUnit.NANOSECONDS);

-- Timers ----------------------------------------------------------------------

test.JSONParserTest.StockPrice Batch Parse.timer

             count = 34356

         mean rate = 95.49 calls/second

     1-minute rate = 95.70 calls/second

     5-minute rate = 88.47 calls/second

    15-minute rate = 80.06 calls/second

               min = 9.32 milliseconds

               max = 244.26 milliseconds

              mean = 10.46 milliseconds

            stddev = 2.38 milliseconds

            median = 10.07 milliseconds

              75% <= 10.64 milliseconds

              95% <= 11.99 milliseconds

              98% <= 13.57 milliseconds

              99% <= 22.14 milliseconds

            99.9% <= 31.03 milliseconds 

SCTP vs UDT

When need a better messaging protocol for my project DTCrawler. I find these 2 new implementation. After study them, especially the book Networks for Grid Applications I choose UDT

1. built upon the UDP (which is my preference) 

2. Provide the flow/congestion management, which it is necessary for the application  

The difference is like this. 

"UDT borrows the messaging and partial reliability semantics from SCTP. However, SCTP are specially designed for VoIP and telephony, but UDT targets general purpose data transfer. UDT unifies both messaging and streaming semantics in one protocol."

Java Performance Tuning Study Memo

Wonderful blogs from http://java-performance.info/!!! List of articles http://java-performance.com/

Java type memory usage

byte, boolean	1 byte
short, char	2 bytes
int, float	4 bytes
long, double	8 bytes

Byte, Boolean	16 bytes
Short, Character	16 bytes
Integer, Float	16 bytes
Long, Double	24 bytes

EnumSet, BitSet	1 bit per value
EnumMap	4 bytes (for value, nothing for key)
ArrayList	4 bytes (but may be more if ArrayList capacity is seriously more than its size)
LinkedList	24 bytes (fixed)
ArrayDeque	4 to 8 bytes, 6 bytes on average

JDK collection	Size	Possible Trove substitution	Size
HashMap	32 * SIZE + 4 * CAPACITY bytes	THashMap	8 * CAPACITY bytes
HashSet	32 * SIZE + 4 * CAPACITY bytes	THashSet	4 * CAPACITY bytes
LinkedHashMap	40 * SIZE + 4 * CAPACITY bytes	None
LinkedHashSet	32 * SIZE + 4 * CAPACITY bytes	TLinkedHashSet	8 * CAPACITY bytes
TreeMap, TreeSet	40 * SIZE bytes	None
PriorityQueue	4 * CAPACITY bytes	None

All Java objects start with 8 bytes containing service information like object class and its identity hash code (returned by System.identityHashCode method). Arrays have 4 more bytes (one int field) containing array length. It looks like all user-written (not JDK classes) have another reference to object Class. These fields are followed by all declared fields. All objects are aligned by 8 bytes boundary. All primitive fields must be aligned by their size (for example, chars should be aligned by 2 bytes boundary).Object reference (including any arrays) occupy 4 bytes. What does it mean for us? In order to get most use of available memory, all object fields must occupy N*8+4 bytes (4, 12, 20, 28 and so on). In this case 100% memory will contain useful data.

Java Boxing Type Caching

Byte, Short, Long	Character	Integer	Float, Double
From -128 to 127	From 0 to 127	From -128 to java.lang.Integer.IntegerCache.high or 127, whichever is bigger	No caching

Java Performance Tips

Never use exceptions as return code replacement or for any likely to happen events (especially in not IO-bound methods!). Throwing an exception is too expensive - you may experience 100 times slowdown for simple methods.

Throwing an exception in Java is a very slow operation. Expect that throwing an exception costs you something between 100 and 1000 ticks in most cases.

Case Study

http://java-performance.info/use-case-optimizing-memory-footprint-of-read-only-csv-file-trove-unsafe-bytebuffer-data-compression/

MYSQL Tips

1. Enable Log

SET GLOBAL log_output = 'TABLE';
SET GLOBAL general_log = 'ON';
SELECT * FROM MYSQL.GENERAL_LOG ORDER BY EVENT_TIME DESC LIMIT 100;