FW: Pendragon "clarifies" Sun's Caffeinmark Claims...

["David E. Y. Sarna" <David@ObjectSoftCorp.com>]

  People with unclean hands should not throw stones.
  
  > > Fresh from sending the lawyers in on Microsoft and accusing Bill
  > > Gates's men of breaking the Java JDK guidelines, Sun now stands
  > > accused of cheating Java benchmarks. 
  > > 
  > > Sun accused of cheating on their Java benchmarks: check out
  > > http://www.zdnet.com/zdnn/content/zduk/1105/206023.html.
  > > 
  > > 
  > > 	This is the latest from Pendragon Software, owner of Caffeinmark
  > > for benchmarking Java compilers. See the underlined sections for
  > > highlights. Seems like a great opportunity to position our VM as the
  > > fastest considering Sun trying to "fix" the benchmark tool!
  > > 
  > > 	The following press release was issued yesterday:
  > > 
  > > 	http://www.pendragon-software.com/pr1197-2.html
  > > 
  > > 	Pendragon Software Clarifies Solaris CaffeineMark Claims
  > > 
  > > 	Analysis of test results for Sun Microsystems latest Java
  > > compiler shows
  > > 	that the compiler specifically scans for Pendragon Software's
  > > benchmark
  > > 
  > > 	Libertyville, Ill., November 4th, 1997 - Pendragon Software
  > > Corporation,
  > > 	the developer of the CaffeineMark Java performance benchmark,
  > > today
  > > 	presented evidence that a Sun Microsystems Java compiler
  > > specifically
  > > 	identifies part of the CaffeineMark test and produces a
  > > misleading score.
  > > 
  > > 	On October 20th, 1997, Sun Microsystems issued a press release
  > > stating
  > > 	that its Web-enhanced Solaris operating environment delivers the
  > > world's
  > > 	fastest Java technology performance. The release also claimed
  > > that the
  > > 	new Solaris Just-In-Time (JIT) compiler held a new speed record,
  > > 50%
  > > 	faster than the highest Windows NT score, as measured by the
  > > CaffeineMark
  > > 	benchmark. Sun provided Pendragon Software Corporation with
  > > benchmark
  > > 	results to substantiate their claim.
  > > 
  > > 	Analysis by software engineers at Pendragon Software showed that
  > > in one of
  > > 	the 9 CaffeineMark benchmark modules, the Logic test, the
  > > Solaris JIT
  > > 	compiler scored 50 times higher than any other system to date.
  > > There were
  > > 	three possible explanations for this result: either the compiler
  > > was
  > > 	performing some impressive optimizations, there was a bug in the
  > > compiler,
  > > 	or there was a bug in the benchmark.
  > > 
  > > 	To determine if there was a flaw in the compiler or a flaw in
  > > the
  > > 	benchmark, Pendragon Software created an alternative version of
  > > the
  > > 	benchmark that used a new Logic test. The new Logic test was
  > > generated
  > > 	from Java source code that had superficial syntactic changes,
  > > and had
  > > 	slightly different bytcodes. There was almost no difference in
  > > scores
  > > 	between the original test and the alternate test running in Java
  > > 	environments on Windows 95, Windows NT and MacOS. However, when
  > > the
  > > 	Solaris 2.6 JIT compiler ran the alternate test, its scores were
  > > 300
  > > 	times lower. This suggested that the compiler was pattern
  > > matching
  > > 	against the benchmark, and performing optimizations that do not
  > > work in
  > > 	the general case.
  > > 
  > > 	Further analysis of the compiler showed that it contains a block
  > > of 600
  > > 	bytecodes that exactly match part of the LogicAtom.class file
  > > used in the
  > > 	CaffeineMark benchmark software. Clearly, the compiler looked
  > > for the
  > > 	Logic test module in the CaffeineMark benchmark and produced an
  > > extremely
  > > 	high score. This gave the compiler a very good overall
  > > CaffeineMark score
  > > 	that would not translate into good Java performance in the
  > > general case.
  > > 
  > > 	On October 29th, Sun Microsystems removed the JIT compiler from
  > > their
  > > 	download page. However, Sun Microsystems' original press release
  > > entitled
  > > 	"Sun's Solaris Runs Java Applications 50 Percent Faster than
  > > Windows NT"
  > > 	was widely distributed, and is still posted at Sun Microsystems
  > > Web site.
  > > 
  > > 	"Though we made a request, no retraction of the original press
  > > release was
  > > 	issued by Sun Microsystems, so we felt it would be fair to all
  > > Java
  > > 	technology vendors to set the record straight," said Ivan
  > > Phillips,
  > > 	President of Pendragon Software. "The fastest overall
  > > CaffeineMark 3.0
  > > 	scores we have seen to date are from Windows NT systems running
  > > on Intel
  > > 	Pentium II processors at 300 MHz."
  > > 
  > > 	The CaffeineMark benchmark can be run directly over the internet
  > > at
  > > 	Pendragon Software's Web site at
  > > http://www.pendragon-software.com.
  > > 	Pendragon Software makes the CaffeineMark software available at
  > > no cost
  > > 	via the Web site, and periodically releases submitted results.
  > > Pendragon
  > > 	Software does not certify the accuracy of CaffeineMark scores
  > > quoted by
  > > 	other individuals and organizations unless specifically noted.
  > > 
  > > 	Pendragon Software Corporation is a software development firm
  > > based in
  > > 	Libertyville, Illinois. Pendragon Software specializes in
  > > software for
  > > 	emerging technologies including Java, the 3Com® PalmPilot
  > > platform, and
  > > 	embedded systems. Pendragon Software can be contacted at
  > > 847-816-9660; fax
  > > 	at 847-816-9710; e-mail to info@webfayre.com; or through
  > > Pendragon
  > > 	Software's Web site at http://www.pendragon-software.com.
  > > 
  > > 	###
  > > 
  > > 	CaffeineMark is a trademark of Pendragon Software Corporation.
  > > Sun, Sun
  > > 	Microsystems, Java and Solaris are trademarks or registered
  > > trademarks of
  > > 	Sun Microsystems, Inc. 3Com is a registered trademark, and
  > > PalmPilot is a
  > > 	trademark of 3Com corporation or its subsidiaries. All other
  > > brands and
  > > 	product names may be trademarks or registered trademarks of
  > > their
  > > 	respective holders.
  > > 
  > >
  Regards,
  David E. Y. Sarna       davids@objectsoftcorp.com
  ObjectSoft Corp. (NASDAQ:OSFT)    http://www.objectsoftcorp.com
  433 Hackensack Ave., Hackensack, NJ 07601
  Tel.: (201) 343-9100    Fax: (201) 343-0056
  >