[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: FW: Pendragon "clarifies" Sun's Caffeinmark Claims...
As specified in the zdnet document:
> Analysis by software engineers at Pendragon Software showed that
> in one of
> the 9 CaffeineMark benchmark modules, the Logic test, the
> Solaris JIT
> compiler scored 50 times higher than any other system to date.
9 modules are part of the test. If the alleged "dirty hands" were NOT
involved, that would bring the performance down BY about 10% , assuming
linearity. Does not make much sence that somebody at Sun would make a
delibarate decision like this to get a 10% advantage.
David E. Y. Sarna wrote:
>
> People with unclean hands should not throw stones.
>
> > > Fresh from sending the lawyers in on Microsoft and accusing Bill
> > > Gates's men of breaking the Java JDK guidelines, Sun now stands
> > > accused of cheating Java benchmarks.
> > >
> > > Sun accused of cheating on their Java benchmarks: check out
> > > http://www.zdnet.com/zdnn/content/zduk/1105/206023.html.
> > >
> > >
> > > This is the latest from Pendragon Software, owner of Caffeinmark
> > > for benchmarking Java compilers. See the underlined sections for
> > > highlights. Seems like a great opportunity to position our VM as the
> > > fastest considering Sun trying to "fix" the benchmark tool!
> > >
> > > The following press release was issued yesterday:
> > >
> > > http://www.pendragon-software.com/pr1197-2.html
> > >
> > > Pendragon Software Clarifies Solaris CaffeineMark Claims
> > >
> > > Analysis of test results for Sun Microsystems latest Java
> > > compiler shows
> > > that the compiler specifically scans for Pendragon Software's
> > > benchmark
> > >
> > > Libertyville, Ill., November 4th, 1997 - Pendragon Software
> > > Corporation,
> > > the developer of the CaffeineMark Java performance benchmark,
> > > today
> > > presented evidence that a Sun Microsystems Java compiler
> > > specifically
> > > identifies part of the CaffeineMark test and produces a
> > > misleading score.
> > >
> > > On October 20th, 1997, Sun Microsystems issued a press release
> > > stating
> > > that its Web-enhanced Solaris operating environment delivers the
> > > world's
> > > fastest Java technology performance. The release also claimed
> > > that the
> > > new Solaris Just-In-Time (JIT) compiler held a new speed record,
> > > 50%
> > > faster than the highest Windows NT score, as measured by the
> > > CaffeineMark
> > > benchmark. Sun provided Pendragon Software Corporation with
> > > benchmark
> > > results to substantiate their claim.
> > >
> > > Analysis by software engineers at Pendragon Software showed that
> > > in one of
> > > the 9 CaffeineMark benchmark modules, the Logic test, the
> > > Solaris JIT
> > > compiler scored 50 times higher than any other system to date.
> > > There were
> > > three possible explanations for this result: either the compiler
> > > was
> > > performing some impressive optimizations, there was a bug in the
> > > compiler,
> > > or there was a bug in the benchmark.
> > >
> > > To determine if there was a flaw in the compiler or a flaw in
> > > the
> > > benchmark, Pendragon Software created an alternative version of
> > > the
> > > benchmark that used a new Logic test. The new Logic test was
> > > generated
> > > from Java source code that had superficial syntactic changes,
> > > and had
> > > slightly different bytcodes. There was almost no difference in
> > > scores
> > > between the original test and the alternate test running in Java
> > > environments on Windows 95, Windows NT and MacOS. However, when
> > > the
> > > Solaris 2.6 JIT compiler ran the alternate test, its scores were
> > > 300
> > > times lower. This suggested that the compiler was pattern
> > > matching
> > > against the benchmark, and performing optimizations that do not
> > > work in
> > > the general case.
> > >
> > > Further analysis of the compiler showed that it contains a block
> > > of 600
> > > bytecodes that exactly match part of the LogicAtom.class file
> > > used in the
> > > CaffeineMark benchmark software. Clearly, the compiler looked
> > > for the
> > > Logic test module in the CaffeineMark benchmark and produced an
> > > extremely
> > > high score. This gave the compiler a very good overall
> > > CaffeineMark score
> > > that would not translate into good Java performance in the
> > > general case.
> > >
> > > On October 29th, Sun Microsystems removed the JIT compiler from
> > > their
> > > download page. However, Sun Microsystems' original press release
> > > entitled
> > > "Sun's Solaris Runs Java Applications 50 Percent Faster than
> > > Windows NT"
> > > was widely distributed, and is still posted at Sun Microsystems
> > > Web site.
> > >
> > > "Though we made a request, no retraction of the original press
> > > release was
> > > issued by Sun Microsystems, so we felt it would be fair to all
> > > Java
> > > technology vendors to set the record straight," said Ivan
> > > Phillips,
> > > President of Pendragon Software. "The fastest overall
> > > CaffeineMark 3.0
> > > scores we have seen to date are from Windows NT systems running
> > > on Intel
> > > Pentium II processors at 300 MHz."
> > >
> > > The CaffeineMark benchmark can be run directly over the internet
> > > at
> > > Pendragon Software's Web site at
> > > http://www.pendragon-software.com.
> > > Pendragon Software makes the CaffeineMark software available at
> > > no cost
> > > via the Web site, and periodically releases submitted results.
> > > Pendragon
> > > Software does not certify the accuracy of CaffeineMark scores
> > > quoted by
> > > other individuals and organizations unless specifically noted.
> > >
> > > Pendragon Software Corporation is a software development firm
> > > based in
> > > Libertyville, Illinois. Pendragon Software specializes in
> > > software for
> > > emerging technologies including Java, the 3Com® PalmPilot
> > > platform, and
> > > embedded systems. Pendragon Software can be contacted at
> > > 847-816-9660; fax
> > > at 847-816-9710; e-mail to info@webfayre.com; or through
> > > Pendragon
> > > Software's Web site at http://www.pendragon-software.com.
> > >
> > > ###
> > >
> > > CaffeineMark is a trademark of Pendragon Software Corporation.
> > > Sun, Sun
> > > Microsystems, Java and Solaris are trademarks or registered
> > > trademarks of
> > > Sun Microsystems, Inc. 3Com is a registered trademark, and
> > > PalmPilot is a
> > > trademark of 3Com corporation or its subsidiaries. All other
> > > brands and
> > > product names may be trademarks or registered trademarks of
> > > their
> > > respective holders.
> > >
> > >
> Regards,
> David E. Y. Sarna davids@objectsoftcorp.com
> ObjectSoft Corp. (NASDAQ:OSFT) http://www.objectsoftcorp.com
> 433 Hackensack Ave., Hackensack, NJ 07601
> Tel.: (201) 343-9100 Fax: (201) 343-0056
> >