Looks like LightZone scans the entire file system


Posting rules: It shouldn't need saying, but... play nice. Please keep your discussions civil. You can disagree, just don't be disagreeable. And, of course, all of the usual stuff like no spamming. Tex adds: I'll be rigorously enforcing this as we go along. We're probably going to be a small community in a little lifeboat, so we can't have members at each others' throats. This is for the sake of the project as a whole. So when you post, pretend you're speaking in person with your very wealthy auntie who has always treated you wonderfully and currently lists you prominently in her will. I won't be tossing anyone out of the forums because we are all in this together (except spammers: immediate membership cancelation), but I'll delete suspect posts right away.


16 posts / 0 new
Last post
jphartmann
Looks like LightZone scans the entire file system

When I start light zone with a particular NFS file system mounted, lightZone goes unresponsive and the NFS server (which is slow) peaks out, so presumably lightZone is mapping out the entire directory structure of my file system.

I did not ask for this action and I need it supressed.

Presumably it is a "favour" deep in the bowels of a java class to be nice and responsive.  It has the opposite effect.

Any suggestions?

 

 

larshenrikoern
It is the normal behavior

Sorry, but that is the default behavior to scan the folder you worked in the last time LighyZone was used. Try to go to a small folder another place on your system, with just a few pictures in it, before leaving Lightzone. I do not have a NFS setup, so it might as well be due to other problems. On what platform are you (linux, Mac, Windows)

 

Have a nice day

jphartmann
I'm on Linux, Ubuntu to be

I'm on Linux, Ubuntu to be more precise.  The folder I work in is highlighted in black in the left-hand frame, right?

That particular directory is /nfs/zfs/data/images/xxx.  It has four files in it, which are shown in the thumbview pane below the main window.

The NFS server being scanned is mounted as /nfs/sfs.

ls (list files) of just one directory of that server takes about a minute elapsed, so there are some long delays when enumarating its directory structure. 

The scan kicks in with a bit of delay after I start LightZone (seconds) and from then on LightZone is unresposive until I kill it from another command prompt.  That is, I cannot even close it with the red x button (since it is hanging the process in a system call).

lsof sometimes shows up to twenty open handles into /nfs/sfs.  The server which is something called z/VM (which I bet you never heard) of is maxed out.  (In fairness it was never designed to parttake in NFS--I'm not sure there was NFS when it was designed.)

I expect LightZone always does this scan, it is just that it is not slowed down by other NFS servers.  But it sure generates a lot of traffic and the information it gathers is of course of no use whatsoever.

Like I said earlier, it is likely something in the bowels of Java that you never noticed.

 

jphartmann
I unmounted the /nfs/sfs file

I unmounted the /nfs/sfs file system yesterday.  When I now start LightZone, all  is fine.  If I click on the root of that file system, I'm told to pick a folder that has images in it.  I get the same response if I try to expand the directory, but then the plus disappears indicating (correctly) nothing in the mount point.  Perhaps that is a hint?

I do have a lot of raw files and LightZone is the only thing I have come across on Linux that can handle that many files with anything remotely like performance, so for me it is the only game in town.

larshenrikoern
Maybe a work around

Hi I am on Linux myself ;-) (gentoo)

 

I will not go into your setup, because as I wrote I have never used a NFS setup myself. But I am very sure it is a good storage solution. But as workaround you might copy the  files you want to work with to a local folder in your home directory. After processing moving them back to your NFS. Hopefully yoor image viewer supports RAW. If not look at Shotwell or Geeqie. And LightZone is indeed one of the most capable photo editing tools out there.

 

Have a nice day

jphartmann
Lars-henrik, we're speaking

Lars-henrik, we're speaking terrabytes of data.  You don't just stick them casually in your home directory.

Sounds like the underlying library does something nobody noticed before.  If anyone can give me a pointer to where I should start looking, that would be fine.

jphartmann
Anyhow, the problem is not

Anyhow, the problem is not the raw files, but it occurs when a completely unrelated file system is mounted.

larshenrikoern
Hi again

Sorry that I asked in the wrong direction. I thought the problem was with reading RAW and working with RAW-files on your NFS. But if it is happening without you are using LightZone for working directly on the files on your NFS, copying files somewhere else is offcourse not a solution.

 

What version of Ubuntu are you using ?? What is the relevant part of the output when starting LightZone from the commandline. It might be a bug somewhere else (maybe systemd or dbus) wich Ubuntu relies on (for the same reasons I am not using them on Gentoo most of the time).

 

Have a nice day

jphartmann
What I am saying is that

What I am saying is that LightZone explores parts of the file system it should not be in unless I open up the path in the left-hand pane.

I have been saying it the time.  Will it help if I do so in Danish?  German?  French?

larshenrikoern
Is your NFS setup correctly

Please dont be rude. I am trying to help you with just minimal information from your side. Have you set up your NFS following this guide https://help.ubuntu.com/community/SettingUpNFSHowTo

I have never before experienced or heard of LightZone having this behaviour before. So I am quite sure it has something to do with your setup. Have you linked (with ln -s )from your (empty) workingdir in your homefolder to the relevant part of your NFS setup. This might be your problem. And I know the history of mainframes (and read in the past about z/vm. And when i searced the following came up https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667828

This might as well be the reason for your slow NFS

 

Have a nice day

Jacal
Yes, please,

don't be rude. I am too Linux-illiterate to try to help here, @larshenrikoern is doing his best. Let's keep this a nice place to be.

jphartmann
Jacal, I'm not rude.  You

Jacal, I'm not rude.  You would not notice it if I were rude.

I am however quite frustrated. 

I state the problem with LightZone quite clearly, but it seems not to be understood.

Lars Henrik states "Sorry, but that is the default behavior to scan the folder you worked in the last time LighyZone was used."

That is fine, but LightZone demonstrably scans the *entire* file system, which is not so good.  And that scan causes the LightZone process to lock up (likely on the order of hours) when a particular file system is mounted, despite the fact that the file system in question is mounted outside the path that contains images.

Chances are that nobody is aware of this because it likely happens deep in a Java class (FileSystem, perhaps?) and it might not be noticeable in the "normal" way of things; certainly not if your PC is not networked.

I am not asking for "help" with my setup.  I've had that for fifteen years at least.  I've worked with the various operating systems involved between thirty (UNIX) and forty+ years (VM).  In short, I do understand what is going on as far as it is observable with UNIX and other tools.    

larshenrikoern
Well then you know best

Well. I am not a professionel computer adminstrator. I am just a volunteer working with computers as a hobby for more than 30 years. So you will offcource know more than I do (or probably anyone here on the site). But I could maybe suggest to use lightzone as a separate user with rights to read (and write) just the needed directories. If this does not work, it must be a serious bug in a library, probably not directly related to LightZone. Starting from the commandline will in most cases give an idea about what is broken, so we can fill in a bug report in the right place. Thats why I asked for the commandline output. But if you are working on a mainframe it is most probably comercial and I do understand that you have to hide your setup. And as you know more than me I might not be able to help that much. Sorry for the inconvenience.

 

Have a nice day

 

sfink16
Failure

Here is an old thread with a member asking questions on NFS --  http://lightzoneproject.org/lorum/cannot-write-mounted-nfs-partition

 

This user's experience ended in failure.  Is it possible that NFS is not supported in LZ?  Is it possible that servers in general cause issues in LZ?  I don't know the answers but perhaps someone else does.  For what it's worth LZ describes it's software in the about section like this --> "LightZone is professional-level digital darkroom software for Windows, Mac OS X, and Linux...".  It never mentions servers on file systems in the about section.

 

Steve

jphartmann
Here is the mangled console

Here is the mangled console when SFS is not mounted.  No doubt what you would expect (apart from the deleted line ends).

[/home/john] lightzone Starting LightZone version 4.1.6 ... with options : java version "1.7.0_121" OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.14.04.3) OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode) This is LightZone 4.1.6 () Running Java version 1.7.0_121 Verifying Libraries Initializing Engine Utilities Max Memory:   14978383872 Total Memory:   504889344 Running on 4 processors Allocating 0MB for the image cache. Loading Classes Scanning Color Profiles Restoring Preferences Starting Logging Scanning Saved Documents Opening... Light Crafts JAI Library - 02/05/07 [/home/john] mount /nfs/sfs

When the SFS file system is  mounted, it displays the same messages as above (surprise!) and puts up the image browser and I can poke around for, say 15 seconds (perhaps while LightZone inspects my audio files).

Then I see:

EventThread blocked for 2.000 seconds EventThread blocked for 4.001 seconds EventThread blocked for 6.001 seconds EventThread blocked for 8.002 seconds

This no doubt goes on forever until LightZone has finished with the SFS file system.

The advantage of using a command prompt is that I can kill it with ctl-c, so thanks for that prod.

 

jphartmann
When I ctl-c I get Java

When I ctl-c I get Java messages that look like listing the status of the various threads.  Perhaps someone can find a clue in there?

Signal Dispatcher: RUNNABLE
Thread-0: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at sun.print.UnixPrintServiceLookup$PrinterChangeListener.run(UnixPrintServiceLookup.java:861)
AWT-EventQueue-0: RUNNABLE
        at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
        at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
        at java.io.File.exists(File.java:813)
        at sun.awt.shell.ShellFolder.getShellFolder(ShellFolder.java:242)
        at javax.swing.filechooser.FileSystemView.getFiles(FileSystemView.java:490)
        at com.lightcrafts.ui.browser.folders.FolderTreeNode.updateChildren(FolderTreeNode.java:220)
        at com.lightcrafts.ui.browser.folders.FolderTreeModel.directoryChanged(FolderTreeModel.java:45)
        at com.lightcrafts.ui.browser.folders.FolderTreeModel$1.run(FolderTreeModel.java:32)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:745)
        at java.awt.EventQueue.access$300(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:706)
        at java.awt.EventQueue$3.run(EventQueue.java:704)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:715)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:218)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:133)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:122)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:118)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:110)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
SunTileScheduler0Standard3: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.media.jai.util.WorkerThread.run(SunTileScheduler.java:439)
AWT-XAWT: RUNNABLE
        at sun.awt.X11.XToolkit.waitForEvents(Native Method)
        at sun.awt.X11.XToolkit.run(XToolkit.java:541)
        at sun.awt.X11.XToolkit.run(XToolkit.java:505)
        at java.lang.Thread.run(Thread.java:745)
DestroyJavaVM: RUNNABLE
SunTileScheduler0Standard1: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.media.jai.util.WorkerThread.run(SunTileScheduler.java:439)
SunTileScheduler0Standard0: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.media.jai.util.WorkerThread.run(SunTileScheduler.java:439)
Java2D Disposer: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at sun.java2d.Disposer.run(Disposer.java:145)
        at java.lang.Thread.run(Thread.java:745)
process reaper: TIMED_WAITING
        at java.lang.UNIXProcess.waitForProcessExit(Native Method)
        at java.lang.UNIXProcess.access$500(UNIXProcess.java:55)
        at java.lang.UNIXProcess$4.run(UNIXProcess.java:228)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Laf-Widget fade tracker: TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at org.jvnet.lafwidget.utils.FadeTracker$FadeTrackerThread.run(FadeTracker.java:479)
Image List Poller: TIMED_WAITING
        at java.lang.Object.wait(Native Method)
        at com.lightcrafts.ui.browser.model.ImageListPoller.run(ImageListPoller.java:45)
        at java.lang.Thread.run(Thread.java:745)
Reference Handler: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
AWT-Shutdown: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:296)
        at java.lang.Thread.run(Thread.java:745)
Image Task Queue: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.ui.browser.model.ImageTaskQueue.waitForNotify(ImageTaskQueue.java:137)
        at com.lightcrafts.ui.browser.model.ImageTaskQueue.run(ImageTaskQueue.java:101)
        at java.lang.Thread.run(Thread.java:745)
TileReaper: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at com.lightcrafts.jai.utils.LCTileCache$TileReaper.run(LCTileCache.java:1241)
Finalizer: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Timer-0: TIMED_WAITING
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)
SunTileScheduler0Standard2: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.media.jai.util.WorkerThread.run(SunTileScheduler.java:439)
TimerQueue: WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:209)
        at javax.swing.TimerQueue.run(TimerQueue.java:174)
        at java.lang.Thread.run(Thread.java:745)
DirectoryMonitor.MonitorThread: RUNNABLE
        at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
        at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
        at java.io.File.exists(File.java:813)
        at com.lightcrafts.utils.directory.UnixDirectoryMonitor.hasChanged(UnixDirectoryMonitor.java:78)
        at com.lightcrafts.utils.directory.DirectoryMonitor$MonitorThread.run(DirectoryMonitor.java:184)
Thread-6: WAITING
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.prefs.AbstractPreferences$EventDispatchThread.run(AbstractPreferences.java:1476)
AWT Watchdog: RUNNABLE
        at java.lang.Thread.dumpThreads(Native Method)
        at java.lang.Thread.getAllStackTraces(Thread.java:1640)
        at com.lightcrafts.app.AwtWatchdog.dumpThreads(AwtWatchdog.java:44)
        at com.lightcrafts.app.AwtWatchdog.run(AwtWatchdog.java:78)
FileCacheMonitor: WAITING

        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.lightcrafts.utils.filecache.FileCacheMonitor.run(FileCacheMonitor.java:56)

[/home/john]