When my crawler project is closed to the end. my concern on the performance is heavier. How to measure my system performance? there is no easy answer for it. So many module, so many parameters, most important is all these CAN'T affect the system performance and increase the module complexity.
Even for the JSON parsing function, I spend almost 1 working day to come out the performance measurement and it is just for the unit testing ;( . The result is just like the following.
stockprice (36648000 records) elapsed ms:376806.531 for 36000 avg:9.937 variance:5.652 Fastest:9.000 Slowest:244.000
[67, 9 x 16910, 10 x 13029, 11 x 4306, 12 x 868, 73, 13 x 248, 14 x 123, 15 x 49, 17 x 15, 16 x 33, 19 x 9, 18 x 8, 21 x 26, 20 x 11, 23 x 51, 22 x 47, 25 x 43, 24 x 41, 27 x 39, 26 x 38, 29 x 24, 28 x 28, 31 x 3, 30 x 18, 34, 35 x 2, 32 x 2, 33 x 6, 38, 39 x 2, 36 x 6, 37 x 3, 42 x 2, 43, 41, 50, 48, 54, 244]
But how about other functions.... I was almost frightened by the future workload. It seems my system launch day need be postpone. Until today, I find this wonderful library,
Metrics, through the netty example. It is fantastic and save me huge time on the performance measurement and reporting.
With just few lines, the following result will be automatically printed into the System console. If you need, it can easily output the result into CSV, log file, JMX, even provide the servlet to remotely pass the result as JSON. Wonderful!!!
With all these tools, I almost got the insurance on my system quality.
final ConsoleReporter reporter = ConsoleReporter.forRegistry(Metrics.defaultRegistry())
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS)
.build();
reporter.start(1, TimeUnit.MINUTES);
Timer timer = Metrics.newTimer(this.getClass(),"StockPrice Batch Parse","timer",new SlidingWindowReservoir(nMax));
timer.update(stopwatch2.stop().elapsed(TimeUnit.NANOSECONDS),TimeUnit.NANOSECONDS);
-- Timers ----------------------------------------------------------------------
test.JSONParserTest.StockPrice Batch Parse.timer
count = 34356
mean rate = 95.49 calls/second
1-minute rate = 95.70 calls/second
5-minute rate = 88.47 calls/second
15-minute rate = 80.06 calls/second
min = 9.32 milliseconds
max = 244.26 milliseconds
mean = 10.46 milliseconds
stddev = 2.38 milliseconds
median = 10.07 milliseconds
75% <= 10.64 milliseconds
95% <= 11.99 milliseconds
98% <= 13.57 milliseconds
99% <= 22.14 milliseconds
99.9% <= 31.03 milliseconds
