tag:blogger.com,1999:blog-7006618857347978140.post3358580197097995994..comments2010-04-01T00:25:54.831-07:00Kevin Lawtonhttp://www.blogger.com/profile/03442192017196947120noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-7006618857347978140.post-74675762111592512642009-05-23T15:09:30.587-07:002009-05-23T15:09:30.587-07:00Hi,<br /><br />I wouldnt like to comment your concerns about power consumption but I'd like to contribute with some ideas.<br /><br />1. If you consider RTSJ (Real Time System Java) you could use ITC (Initialization Time Compilation) instead of JIT. RTSJ can speed up your Java application too if you use "Soft Real Time Threads", which is not difficult to implement and can prevent GC to manage memory you can manage easily yourself (Scoped Memory).<br /><br />These links may be of your interest:<br /><br />* http://java.sun.com/javase/technologies/realtime/reference/doc_2.1/release/JavaRTSCompilation.html<br /><br />* http://www.rtsj.org/specjavadoc/book_index.html<br /><br />2. IBM has a very interesting research project called X10 which generates code in Java and/or C++ as output. The input language is something based on Scala (see release 1.7.x).<br />You could use it to Write-Once- Run-Everywhere, does not matter if you have a JVM or your your native OS.<br /><br />A very interesting improvement over Scala is that X10 does not use MPI but it uses PGAS, which is beneficial as STM but provides maximum performance for local data.<br /><br />IBM X10 Language<br />* http://www.x10-lang.org/<br />* http://dist.codehaus.org/x10/documentation/languagespec/x10-173.pdf<br /><br />STM (Software Transactional Memory)<br />* http://en.wikipedia.org/wiki/Software_transactional_memory<br /><br />PGAS (Partitioned Global Address Space)<br />* http://en.wikipedia.org/wiki/Partitioned_global_address_space<br /><br />Regards<br /><br />Richard Gomes<br />http://www.jquantlib.org/index.php/User:RichardGomesrgomes1997http://www.blogger.com/profile/10993828277309610363noreply@blogger.comtag:blogger.com,1999:blog-7006618857347978140.post-29989247207273138962009-05-13T06:20:00.000-07:002009-05-13T06:20:00.000-07:00(I thought this was intriguing enough, and was proud enough of my reply on this to the mahout-dev list, that I will favor you with a cross post here!)<br /><br />The difference in power consumption between a fully loaded machine and<br />idle isn't so large (the figure 50% sticks in my head?), but the<br />difference between a fully loaded and half-loaded machine is quite<br />small. That is, if the hard disk is up, processor is at full speed,<br />all memory is fully powered, then using all or most is not a big deal.<br />Power consumption drops only if you are really idle.<br /><br />I don't have numbers to back this up at my fingertips, though they're<br />informed by figures I've seen in the past. I think that's what one<br />would need to evaluate this argument, and I have a different intuition<br />about how much this could matter.<br /><br />The main argument here seems to be, basically, that Java competes well<br />in wall-time performance by better parallelism and more memory usage.<br />Maybe, that's an interesting question. Is LLVM going to be more<br />efficient than Java? unclear, both have an overhead I suppose. But<br />again interesting question.<br /><br /><br />But, the topic really does matter. Wasting time means wasting energy,<br />and when we get to distributed cluster scale, it matters to the<br />environment. At Google they do a good job of keeping teams really<br />clear about how much their operations are costing -- it is staggering<br />sometimes. Developers who might run a big job, oops, see it fail,<br />start it up again, oops, wrong argument again... might think twice<br />when the realize how many pounds of CO2 their mistake just pumped into<br />the atmosphere.<br /><br />(Mahout folks will now appreciate why I have been messing with the<br />code all over to try to micro-optimize for performance. I think there<br />is still not enough attention given to efficiency yet, but hey it's at<br />0.1.)<br /><br /><br />And, I think I agree with the conclusion of the blog post for a<br />different reason:<br /><br />The Java/C++ performance gap for most apps is pretty negligible these<br />days. Why? I actually think given a fixed amount of *developer* time,<br />one can make a faster Java app than C++ app. Why? I can develop<br />faster, against a larger and more stable collection of libraries,<br />spend less time debugging, leaving more time to optimize the result.<br /><br />But that does hit a certain plateau. Given enough developer time, I<br />can get native code to run faster than even JITted Java. I myself am<br />hard-pressed to optimize my code (Mahout - Taste) further in Java<br />without drastic measures.<br /><br />It may take a lot of time to actually beat Java performance in C++,<br />but, as the scale of your operations grows, the return on that 1%<br />improvement you eke out grows. And of course -- when we talk about<br />code headed for Hadoop, we are definitely talking about large-scale<br />operations.<br /><br />For reference, of course, Google operates at such a scale that they<br />use a C++-based MapReduce framework. It is just almost always<br />worthwhile to spend the time to beat Java performance.<br /><br />This isn't going to be true of all users of distributed computing<br />frameworks, so it's not inherently wrong that Hadoop is in Java, but,<br />I did find myself saying "hmm, Java?" the first time I heard of<br />Hadoop.<br /><br /><br />But isn't this what this whole Hadoop streaming business is about?<br />letting you farm out the computation itself to whatever native process<br />you like and just using Hadoop for the management? because that of<br />course is fine.srowenhttp://www.blogger.com/profile/06524814758673314736noreply@blogger.com