An interesting development in the C-based behavioral synthesis space is the development of a benchmark suite. CHStone is a proposed suite for providing a common platform of benchmarks upon which to compare high-level synthesis tools. It's an interesting idea -- as it might be a way for potential customers to assess, compare and contrast different tools, without having to devote as much work. As well, if the suite is really representative, realistic and broad enough in capability, then it will allow customers to understand how the tools might perform in different applications.
At a minimum, it would be another datapoint prospective customers could use. Wouldn't it be interesting if vendors published source code, any support files required by the tool, generated Verilog RTL, and synthesis results (for popular silicon targets, e.g. TSMC 65 nm, Xilinx, Altera, ...) for each design in the benchmark suite?
People using published results should be thinking about an array of considerations:
* What's the source code like and ancillary files, if any
* How usable is the RTL (debug/ECOs/...)
* Can end-users recreate the results
* In the microprocessor space, compilers and processors were tuned to address the benchmark suites. For example, there were compilers that recognized that the Dhyrstone source and spit out a pre-designed, hand-optimized assembly code. Other tricks like optimizations that are narrowly tailored such that the benchmark(s) benefit but other designs almost never do. It's likely that games will be played over time if people start using these designs -- so, end-users will have to change the sources in different dimensions to test for this (features/naming/loop structure/....). And, this may diminish the value of this suite over time -- so new suites will need to get developed.
* I'm sure there are some others... (I've got to run -- I'll probably add some more when I've had time to digest.)
I don't have access to the IEEE paper that the developers of the CHStone suite wrote -- I'd love to read it to understand more. I think it's an interesting idea, as long as it doesn't get gamed. I'm curious about how good the proposed designs are at really pushing the tools -- and pushing them in different directions. Thoughts?
Thursday, February 05, 2009
Subscribe to:
Post Comments (Atom)
2 comments:
Another important metric is target portability. Can the code "travel"? There once was a m-code to gates tool that did fine, so long as one made a significant investment in specializing for a specific target. If those investments are lost when moving to another target; the utility of reuse and abstraction can be severely diluted.
Good point. End users need to be able to assess just these things.
Single data points are not that meaningful for a public benchmark like this (e.g. 74K gates at 354 MHz), without independent reproducibility, the ability to assess all the things it takes to get those results, and the data to be able to understand implications like portability and ease of debug/ECOs, ...
Post a Comment