image reader nodes limited to 4gb for .tif files?
hello knip community,

i am building knime workflows to work with microscopy image data for biological research. the .tif file format is the most widely used in my work, and these files can often get quite large, with the current largest we’re working with around 10gb. i am finding issues trying to load large .tif files using the image reader nodes, and thought i would see if there is something i am not doing right.

i have tried loading the large files using each of the three image factories, and i get the same results, with errors saying:

warn image reader (table) 3:281 encountered exception while reading image: row8! view log for more info.

warn image reader (table) 3:281 encountered errors during execution!

i have tried looking in the error log, and the other log that can be accessed by view > open knime log, and i don’t see any additional information other than these two error message lines. i don’t think it is the heap space or a memory issue, because i am giving knime 20gb of ram to work with, i’ve not seen any error indicating it is a memory issue, and when i watch the memory in my task manager i have found that knime just closes itself if it runs over memory by opening multiple smaller files. smaller files that do load use more memory than the large files when they return an error, so i think the issue is not a memory problem.

i know the node can access the large files, because if i load only metadata i can access the ome-xml in the output. i have tried loading subsets of large image stacks, in multiple combinations of limiting z slices, channels, or time points, and have also tried limiting the series read in, and none of these attempts have changed the result. i have tried both the image reader and image reader (table) nodes, and see the same behavior.

i can load the large images in fiji/imagej without a problem, so i tried opening a large image inside an imagej1 macro node, both with the standard open and with the bio-formats import, and i find that the imagej1 macro node does not seem to be able to open images of any size programmatically from inside the node; i’ll make a different post about this. i also tried the ij image reader (2d) – experimental, and this says it loads the large image file, but only the first plane is accessible.

it seems that the standard .tif file format specification is limited to 4gb, so i tried using a bigtiff format with a .tf2 extension, with no change in the results. i then decided to test this 4gb limit, and sure enough, a 3.93 gb image with a file size of 4.13 gb including the header will load (only with the planar image factory, the others throw errors), but a 4.11 gb image with a file size of 4.31 gb will not load.

so, are the image reader nodes limited to 4gb for .tif files, or have i missed some option that would allow them to open the larger files? it’s not a big deal if so, i can see a workaround for our large files, and i very much appreciate all the beautiful work that has gone into knip that i can just use and build on, i just figured i’d see if there’s something i’m not doing right. additionally, i find that loading the 3.93 gb image file causes knime to use 13 to 15 gb of ram, according to my task manager, another thing that makes me think i may be missing an option somewhere. does this memory usage sound correct, if so what is it doing with all that extra memory? loading the same file in fiji / imagej uses about 4.5 gb.

i look forward to any answers, thoughts, or comments. let me know if there is any further additional info i can provide.

thanks!

jeff
hi [@jeffb](/u/jeffb),

i did some digging: imagej1 by default exports regular tiff files, even if those turn out to be larger than 4g. those files can magically still be read by imagej1, but since are not proper bigtiffs not by other tools. you can replicate this issue easily by trying to import one of your large files via file > import > image in imagej/fiji.

in order to open such files in knime, they have to be proper bigtiffs. you can export them via bio-formats, i.e. plugins > bio-formats > bio-formats exporter, by using the btf file extension for your files. i am not entirely sure if bf has a check built in that would make it a bigtiff even if the file extension were tif. this way, however, i was able to import a 6g btf file with the image reader.

if you want some more details, this issue has them:

[indexoutofboundsexception when reading ij1 tiff hyperstacks (>4gb)](https://github.com/scifio/scifio/issues/317#issuecomment-237833474)

opened 12:22pm - 15 jul 16 utc (utc)

[[imagejan]

imagejan](https://github.com/imagejan)

when using scifio (in ij2 or in knime) to read a large hyperstack that was saved using imagej/fiji, i get the...

best,

stefan
Out of curiosity, what are the sizes of the files you’re trying to move? The file looks to be fine. It's 4500 X 300 pixels and the quality looks decent. 

The size probably that you're trying to print is quite large and you'll be requiring a larger file to avoid pixelation. 

I'd suggest you probably check with your printer the minimum file size required to print such a canvas and then license a file accordingly. it’s a limitation of network shares.

the proper display of jpg thumbnails has a dependency on the media library.   the media library is not created for network shares, only local storage.

some people have said that if you precede the jpg file name with a “.” that the problem will go away.

example… hi ricardo,

the default cache size for the player is 20% of the available ram.

you can use the following environment variable to override the default value (to 4096 mb in this example):

setenv sgc\_pref\_reader\_cache\_max\_size\_in\_mb 4096

i hope this helps.

eric a couple of things to check:

(1) make sure your file realy exists

(2) your file path does not seem to be correct full path, should it be something like c://users/…

(3) you can also try a relative path, namely, copying your tiff file to the same directory as where you start your jupyter. then, use… I'm assuming that your provided image is the actual size that you are dealing with. It appears that the resolution of the image is quite low so that it will suffer pixelization with any enlargement at all. Fidelity of a raster image is always dependent on how much effective resolution (what its resolution is at the size it is to be used) the image has. So a 72 ppi image will only have 36 ppi if it is enlarged to 200%. Use File>Export As and export at 300 ppi or higher and see if that makes a dif... Until you have a computer with no moving parts, hard drives or tapes, you cannot push enough data through a 1 gig card to use it’s full capabilities. The best servers can burst up to about 600 meg. Only a backup could sustain any data transfer that gets close to that. At this time, 10 gig is for switch to switch. image events should be fine, although from memory there may be a bit of an issue with aliases. the main thing is to remember to give it full disk access, otherwise you risk silent failures when it tries to read something where it’s not, by default, allowed. seek and tell are still required.

see here: <https://docs.lvgl.io/latest/en/html/overview/file-system.html#use-drivers-for-images> When designing for a variety of resolutions you want to scale down, not up. Do your design in a @2x resolution (640 x 960 @72ppi) and for the @1x assets you duplicate them to a new document (preferably) and scale them down by 50%. You can set up a Action script for this also to make the process go by faster. To learn a bit about Photoshop action script you can have a look [HERE](https://www.youtube.com/watch?v=BRpPGiGXcHc).


As for your second question, [HERE YOU GO](https://www.dropbox.com/s/sczltvow7jgtfiy/iPhone%204%20GUI%20PSD%20Retina%20Display.psd). The "reload" symbol you were after can be found in that Retina asset sheet, a sheet that is very good to have for anyone who wants to design for iPhone.