After a recent Fuji GF lenses review I posted I have received many comments from people wondering about how I test my lenses and the accuracy of the tests.
As I am fortunate enough to have access to all of the Fuji's lens range I would love to get my tests as accurate as possible, and to be as transparent as possible. So if you have any suggestions about flaws in my test AND how to overcome them, or new tests I could do, I would love to hear them.
For my sharpness tests I have been doing the Bob Atkins method
I have five 6x4 inch prints mounted to a roughly 60x40 inch (1.5x1 meter) board.
One of the charts is placed in the center and one in each corner of the board.
The camera is then set up with one chart in each corner (position is checked on the in camera focus points) of the shot and photographed at each major zoom range (depending on what is marked on the lens for a zoom lens) and each aperture.
I then load the images up on a 4k monitor, zoom in to roughly 800-1100% and measure the different densities of lines per mm that the lenses can resolve.
There are many variables when doing testing like this so below I have added a list of ones I have and have not accounted for.
Light - Gordox SL60W with the same octabox diffuser, at the same measured distance for every lens, set to 100% power every time.
Camera shake - Tripod, 2-sec timer, image stabilizer off if it is an option, electronic shutter, and multiple images taken for each shot just to be safe with the sharpest image measured.
Focus - Manually focused for every single shot. On the center for when measuring the center chart, and on the corners for when measuring the corner charts.
RAW file converters - I noticed a huge difference in results of RAW file performance between different software's (mostly Lightroom classic and Capture One Pro 20) so to remove this as an ever changing variable depending on software updates, I decided to measure all images from the in camera jpegs. This also removes most distortion and aberrations unfortunately but would give me a result that was more consistent and easier for everyone else to replicate. But more importantly would remove another variable making it more consistent to compare lenses.
Exposure - I personally never change metering modes (and didn't for this test). On my camera I used an auto shutter speed, I originally worried about auto lower shutter speeds risking blur, but after testing and with extra attention paid to the camera shake section of variables this turned out to be a unnecessary worry. And with some much consistent light on the charts I never once have had a test image look even slightly different in exposure.
ISO - 100 for GF lenses (GFX 50R), 160 for XF lenses (X-T4)
Daylight WB, Adobe 1998 colour profile (doubt these make any difference but set the same anyway)
Accidents - bumping the camera, small / micro earthquakes, wind shaking the house, pets, etc. After testing many lenses If I ever randomly got a jump or spike in detail when looking at the charts I could put this down to one of these variables and retook the tests. But on this note I never tested lenses during heavy wind or experienced any earthquakes while testing (that I know of), over months of testing I have bumped the camera once or twice, especially in the earlier days of testing, but good practice has reduced this variable frequency.
Not accounted for
Focus distance - This is the biggest one, different for every lens but what I felt was more important was that I wanted my charts to be the same size on the image sensor for every single lens, and in the same position on the sensor as a better way to compare lenses. But lenses may have different results at different distances, and I am unsure how to find out at what distance resolves the most detail, e.g. the further away you get from the chart, the smaller the chart becomes, making it harder for the camera to resolve as much detail (without optically zooming in).
This variable I would like the most feedback on how to overcome, but I need a result that would still give a consistent result for when comparing different lenses e.g. how sharp is a 35mm vs 50mm vs 800mm
1: Size of the board makes it easy to shift around for when needed more space for testing longer lenses e.g. the 100-400mm with a 2x converter.
2: It is an accurate size for what I would call an average distance for most lenses (personal opinion, not sure what is “the average subject distance for every single lens option”)
3: Accurate for closer subjects / foreground objects with wide angle lenses (see con 1)
1: Doesn't give as accurate an indication of sharpness for wide angle lenses of far away subjects.
(Please feel free to inform me of more pros and cons)
Lens Build Variation - production consistency of the lens out of the factory. I could overcome this by testing many of the same lenses over time, but unfortunately this would take far too long and still never promise that someone else would get the same results as me. However when possible I did test multiples of a few lenses especially if the first test gave poor or outstanding results (harder to achieve with very new lenses).
Firmware - I stay very up to date with lens and camera firmware but firmware is a factor that will change over time. The biggest shock here was when testing a XF lens which was performing fantastically, only to remember I hadn't updated the lens firmware part way through the test, and after updating the lens and restarting the test found that the firmware update broke the lens resulting in kind of like a pixelate image, this wasn't noticeable at 100% but it was at 800% zoom, kind of like every second pixel had moved to the left and its empty space was now cloned, needless to say the lens was replaced and re-tested.
Human Error - fortunately I am human, unfortunately we are flawed, I am sure there will be the odd micro jitter caused by me not holding my breath while testing, or maybe I was just in a generous mood, hungry or tired when deciding if a lens resolved between two different densities of lines per mm (at 1100% zoom this is very very fine tooth comb kind of testing), but you get the idea.
If you have any constructive feedback on this process I would love to hear from you, either via social media or by sending me an email, as improving results and gathering more information for my fellow Fuji shooters is a passion I enjoy.
Look forward to hearing from you.
Thank you kindly to those who have made suggestions so far. Most of which has been about the final about of the graphs. When testing XF lenses I thought a simpler way of displaying sharpness was required. As many XF lenses are targeted at those new to photography and I wanted my information to be a gate way to more informed lens decisions. I simplified this by showing the sharpness on a scale of 0-14.
But when it came to GF lenses many of you requested more detail in the graphs, to show more hard definition in what exactly my test results were e.g. no need to dumb it down.
So I changed to Y axis to show the actual lines per mm result I resolved from my testing. Resulting in the below 2.0 graphs
I do need to stress that at 800% zoom the charts are hard to read as the lenses get softer. Trying to work out where exactly the changes in density are when the indicator lines marking the change are a blurry mess them selves. But if I was to guess the margin of error grows to roughly 0.5 lines per mm (lpmm) as once sharpness drops below 2 lpmm. And above 2 lpmm I would guess my margin of error is roughly 0.3 lpmm.
What are your thoughts on my testing? Id love to here from you.