text:
Sayonara !!!
Join Date: Jan 2003
Location: Austin, Texas
Posts: 9,297
you have a whole number of people mentioned here who are of different thoughts it would appear...
what is acceptable to one may not be for the other and who decides which to follow and what information to release to the readership ?
frankly speaking... when it comes to video cards... I don't really care what kyle has to say anymore...
hopefully the inclusion of Dave will add some legitimacy to this 'alliance'
Posts: n/a
Honestly, if AT and [H] are The Big Two as you seem to make them out to be, why are probably the most technical people becoming more and more active at Beyond3D? Why are people from ATI, NVIDIA, Beyond3D, Warp2Search, AnandTech, [H], and Bjorn3D, to name a few, active on these boards, rather than [H]'s or AT's?
The big sites have become homogenized--they run Benchmark X, Benchmark Y, and Benchmark Z, tell you those numbers, and that's it. The little sites, where a reviewer does not have to review a card in ten hours, get a feel for a card, know what it does weird, finds out about bugs and problems in the drivers, and basically has the experience that a normal user would have and then reports it to the public.
That's a Good Thing.
And let's see.. if Company X gives money/hardware/special info to Site Y, what company do you think Site Y would push the group of sites to be more lenient towards? And do you honestly think Site Y would tell you that? Get real. The point of multiple hardware sites is to keep each other in check, not to report the same things over and over.
Posts: n/a
I think NVIDIA themselves have done a very good job at defining legitimate optimizations when they revealed their Optimization Guidelines to reviewers and to the public(although they don't seem to follow them to this day, including the Detonator 51.75 )
I am sure that you are well versed with them, but here they are again:
1) An optimization must produce the correct image.
2) An optimization must accelerate more than just a benchmark.
3) An optimization much not contain any pre-computed states.
You will forgive me if this is not verbatim, but it is close. #1 seems very clear, IQ must not be changed or decreased. #2 is a little tricky, I personally believe the intention here is to make sure that any games with benchmark modes do not have just the benchmark mode optimized, but any optimizations must bleed through to the game itself. Other people have interpreted this to mean NVIDIA thinks synthetic benchmarks should not be optimized at all(which I do not think was NVIDIA's intent). #3 seems pretty straight forward as well, things like inserted clip planes should never be acceptable.
In addition to these guidelines, I think this absolutely needs to be added.
4) An optimization must respect a developer's wishes. ie, when Futuremark says NVIDIA's optimizations are not legal in 3dmark03 they are not legal optimizations.
In regards to the points you bring up:
1) I think an effort should be made to perform an apples to apples comparison as much as possible. If you also want to benchmark an the game with tradeoffs to reflect "realworld" gaming then so be it.
2) Review samples should use either the shipping driver or the latest available driver. If an IHV hands you a driver that will not ship with the product or will not be immediately available to users of the product, it should not be used.
Case in point, with the HL2 benchmarks just last week NVIDIA wanted reviewers to test with Detonator 51.75. This driver shouldn't be used because it is not publically available, nor is it shipping with any product. A second look at the benchmark can always be done later once the driver has been properly released.
It should also be noted that when testing a driver that is either buggy or known to contain cheats, a previous clean driver should be selected if applicable e.g. 51.75 either cheats or is very buggy with regards to IQ(what seems to be a global IQ reduction of games that use PS2.0 as well as globally applying the UT2003 pseudo-trilinear filtering/trilinear AF to all D3D games).
Just my two cents.
Registered User
Join Date: Feb 2003
Posts: 52
Registered User
Join Date: Sep 2003
Posts: 38
Besides that, the AT forums have active members from tech companies like IBM, Microsoft, AMD, ATI, NVIDIA, DFI, OCZ, and Mushkin, to name a few. We also have members from web sites like the [H] (Brent, not Kyle), 2CPU, B3D (Daves not very active, but hes posted several dozen times) and Storage Review, to name a few. Im not an active member of the [H] forums but I lurk there occasionally, so I dont know their situation. But just as I lurk at the [H], Im sure many other companies and web sites lurk at the AT forums. Quite frankly though, a web site doesnt need to be as technical as is absolutely possible to be valuable to end users.
1) Should reviewers use non WHQL drivers?
2) Should reviewers use early stepping silicon when they dont know for sure if itll perform identically to final silicon?
Last edited by Evan Lieb; 09-16-03 at 03:52 AM.
Join Date: Apr 2003
Location: Blue Circle
Posts: 69
I'm pretty sure that nowhere is it stated that texture filtering may only be done on the first tex unit. ATI started this. Now, unfortunately, NVIDIA is doing the same and more. This is why developers are pissed: If I write code that switches on tri-linear, the graphics card has to render it, or it is a bug. Since I do not want to work with buggy cards, I'll switch to a board that does. Too bad there are non left
Posts: n/a
re: trilinear filtering, I don't know if there is anything in the DX spec that says you have to do trilinear on all texture stages. However, IQ certainly IS degraded in UT2003. And there is no way to get normal IQ, therein lies the problem. Even though NVIDIA added in an "Application" setting in 45.23, selecting "Application" does nothing. You still get pseudo-trilinear and pseudo-trilinear AF. 51.75 still is like this.
Join Date: Sep 2003
Posts: 60
Official Nvidia news:
"Due to popular demand by the GFFX users, we have decided that performace is more important than image quality. As are result our image quality from now on will decrease to give the best performance possible. Our cards were, and always will, be the best card to play benchmarks with." -Invidia
Posts: n/a
If NVIDIA really wants to offer users a tradeoff they should do so through the Intellisample slider in the drivers. For example, adding an "Application" setting was supposed to solve the UT2003 trilinear debates. But "Application" does not work, there is still no way to get pre-44.03 UT2003 quality. This situation becomes compounded with comments like this coming from NVIDIA:
1) Since not all IHVs release only WHQL drivers, I would say using only WHQL drivers is impossible. For example, there was no official WHQL driver released for NV30. 43.51 was a beta, unofficial driver and was the first driver to pass WHQL certification for NV3x cards. 44.03 was the first official driver from NVIDIA to pass WHQL, and was launched with NV35. Since users in the past have used official drivers that are not WHQL I don't see a good reason why you shouldn't test non-WHQL drivers. Unless all IHVs make a commitment in the future to only release official drivers that are WHQLed, testing only WHQL drivers might be hard on your part for reasons described above.
Also of interest is the testing methodology required for WHQL. From my understanding DX9 spec adherence is not part of WHQL testing. Which further weakens the necessity of only using WHQL drivers. For example, high dynamic range effects are missing from HL2 on NVIDIA cards because NVIDIA's drivers do not expose float buffers(although the hardware is supposed to support them).
2) This should be rephrased as "should reviewers use early stepping silicon at all." There is no way to know whether or nor final silicon will perform the same or not for sure. You may have assurances from a company that it will, but nothing concrete. I don't see a problem testing early/pre-production silicon. It provides an interesting glimpse of the future. As always, anything that is not final should have a disclaimer indicating such.
Bit Bumper
Join Date: Aug 2002
Location: Here
Posts: 782
Whether or not tri-linear or bi-linear filtering is used is up to the video driver/hardware. In DX, the app just asks for some filtering to be done. The purpose of the design was to allow the video card companies to promote what worked best for the hardware and thus removing the responsibility of the application to determine what should be used and when.
Quite frankly, while it certainly can be done, it would make for a significant amount of code to make the determination for each drawprim and slow the app down.
Doing tri-linear on the first texture stage only is not a bad methodology. Normally the remaining texture stages are various types of maps (cubic, environment, bump....). Very rarely are these later texture stages high detail textures, which is where tri-linear helps the most.
I cannot fault NVidia for disabling tri-linear. If the hardware does not have the power to do full tri-linear, then it is a good call to disable it in order to keep the game playable. I would expect ATI to have some thresholds and disable tri-linear if the frame rates drop too low as well. These are perfectly acceptable things to do and devs expect it.
1) Most every site I have seen appears to be using a frame grab of the front buffer to get an idea of the IQ of the app. Well, there is a problem here. The video card driver is responsible for getting that frame. The frame it CAN grab is not neccessarily the frame buffer, or what is actually being displayed on the screen. NVidia has said that they do some post-processing of the buffer, so the screen shots may not reflect what is really on the screen.
This simply means they are not actually returning the true frame buffer. By defination the frame buffer is the frame being displayed.
Basically what I am getting at, is you have no way of knowing if the actual frame buffer is being returned when a screen shot is taken.
2) The public nature of the benchmarks used by reviewers has allowed video card companies to put in specific hacks/cheats/optimizations (use them interchangeably) to get better results in the benchmark. No amount of posturing by the public/consumer is going to change that. That point should be very clear by know.
3) People are buying into the 'it's ok to optimize,..but/if' argument. Here are some of the problems with that, and it has been very well brought out in this thread. Where is the line? Who determines where the line is? Who checks and verifies the line is not being violated? Who has enough insight to be able to make that call? Does anyone have the right to make that call?
Video cards are big business and while the high end cards are not the lion share of the business, they generate more free press than any other video card product. The board of directors of these companies would be in violation of their fudiciary responsibility, to the investors, if they did not seek out any way possible, to put the company in the best light possible, in terms of what it takes to be recognized as "King of the Hill".
4) Consumers have no idea what should be showing up on the display in an application. Only the developer actually knows what it should look like. This leads reviewers down a path which is to take your "best guess". Just because one card may look better does not mean it is correct.
Well, the above are just covering the issues in a broad sense, but to address a problem, you have to identify it first. Then pick a starting point and get to work.
Solutions could be available to the above issues, but it will take a major mindset change for everyone involved to pursue those solutions. Talking about it is just so much noise. Doing something about it will take time.
I am working on a solution, as I have alluded to in a few posts over the last couple of months. I am not ready to announce what I am doing, but the above is what I am attacking. The problems can be resolved.
Stuff Happenz!
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
You're right, I was misunderstanding.
Yup, I thought you were actually here trying to raise the standards of reviewing as a whole Evan...my mistake, you're here because you realize your credibility is tanked and now you need to add some to your site before it becomes a joke.
Don't try and dictate, try and listen; it could save you.
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
Registered User
Join Date: Sep 2003
Posts: 38
Re: You're right, I was misunderstanding.
|Thread Tools