Too many open files message

[Jerry Jenkins]

Again, all very useful. I think regarding it as an interaction between Publisher and the Max OS may be the key. I have have several machines here and have been work on the one with the older 10.10 operating system to use legacy software that I need for photomicroscopy and scientific work, and to stay out of the cloud as much as possible. My next step will be to upgrade one of the newer machines, or possibly purchase another, and run the Affinity suite on that. From what you all say, that may be the best solution all around. It will take me about a week to switch over and test. I will report back to on this thread once I have.

Psenda, I am sorry that the carriage returns display poorly for you, and won't use them. They look fine on my screen and some naturally to me because I write in short paragraphs. Many things to learn. Thank you all for the help.

J.   

 

 

[R C-R]

1 hour ago, walt.farrell said:

The Affinity applications operate at the normal application level, and I don't know what facilities Mac or Windows provide to a normal application that allow monitoring for file changes without having the file open.

On Macs, when a user launches an app this typically causes up to a few thousand files to be opened by the OS. This is true even if the app does not open any document files. Some are in the application domain, but others are in the system or user domains. Most in the latter two domains are shared with other processes & will remain open as long as any process needs access to them.

In addition to & independently of that, the Mac OS also runs processes that keep track of which files are open, which processes are allowed to read from or write to them, which user 'owns' them, when their contents have changed, & so on. In this context "user" includes not just human users but also ones like root, _spotlight, _appleevents, _windowserver, several daemons, & others.

So there really should be very few situations that cause a 'too many files open' error when running any Mac app. The only one I can think of that would be likely to cause that is running a very large number of apps concurrently, & only if the total open file limit is set to something other than "unlimited."

[Pšenda]

1 hour ago, R C-R said:

so that is not a problem.

No, problem it is not. But it is definitely not the right and good technique to keep files always open.
Fortunately, APublisher doesn't do that either, because it always releases linked files after it detects attributes (detecting possible content changes).

And as you can see, it does this when using fonts (* .ttf files) when rendering UI.

Create - Close, Create - Close, so it should be done 🙂

[Pšenda]

To add - ADesigner does "not detect" a change in the name (loss) of an embedded document, even when you open the Resource Manager!
The embeded document will only be checked when the main document is opened, which surprised me quite.

[walt.farrell]

13 minutes ago, Pšenda said:

To add - ADesigner does "not detect" a change in the name (loss) of an embedded document, even when you open the Resource Manager!

None of the applications detect changes in the original of embedded documents. Changes in the original are irrelevant; the document is embedded, and its contents are entirely contained within the document it's embedded in. Change detection is only for linked documents.

[R C-R]

2 minutes ago, Pšenda said:

No, problem it is not. But it is definitely not the right and good technique to keep files always open.

The Mac OS typically keeps hundreds or even thousands of files open simultaneously, only closing any of them when no process requires access to them. It does this to improve performance by avoiding the needless overhead of constantly making the file system manager open & close files when there is no good reason to devote any system resources to doing that.

This is generally considered to be a 'right and good' file management technique, at least for the Mac OS & other UNIX-like operating systems. I have no idea if anything similar would apply for Windows.

[R C-R]

1 hour ago, Jerry Jenkins said:

My next step will be to upgrade one of the newer machines, or possibly purchase another, and run the Affinity suite on that.

Please let us know if that resolves the 'too many files open' issue for you. Particularly if it does not, this would be something the developers probably would want to know about.

[Pšenda]

36 minutes ago, walt.farrell said:

None of the applications detect changes in the original of embedded documents. Changes in the original are irrelevant; the document is embedded, and its contents are entirely contained within the document it's embedded in. Change detection is only for linked documents.

??? So why does ADesigner "detect" changes in an embedded document, when I open a master document? - if it is completely irrelevant?
Edit: so I - was wrong. Wow, I didn't really expect that (wasn't it in previous versions?).
So why is embedded documents in Resource Manager listed as OK - when it is completely irrelevant?

[walt.farrell]

17 minutes ago, Pšenda said:

So why is embedded documents in Resource Manager listed as OK - when it is completely irrelevant?

It needs to show you all the linked or embedded files, and each needs some kind of status

As for Designer detecting changes in an embedded file there were one or two bugs in that area that were fixed in 1.8 or one of its revisions. 

[Pšenda]

43 minutes ago, R C-R said:

The Mac OS typically keeps hundreds or even thousands of files open simultaneously, only closing any of them when no process requires access to them. It does this to improve performance by avoiding the needless overhead of constantly making the file system manager open & close files when there is no good reason to devote any system resources to doing that.

This is generally considered to be a 'right and good' file management technique, at least for the Mac OS & other UNIX-like operating systems. I have no idea if anything similar would apply for Windows.

One thing is accessing files from an application perspective (we're talking about this here), and the other is managing files and accessing them from an OS perspective. It goes without saying that the OS uses various techniques to optimize access and increase efficiency, such as caching, keeping handles to frequently used files, etc.
n general, however, the application itself should not hold thousands/millions of active connections because it is obvious that it can grow above all limits, which then cause the system resources to overload.

[Pšenda]

3 minutes ago, walt.farrell said:

each needs some kind of status

But if the status is the only one possible state - OK, then it completely loses meaning 🙂
At least I expected the change of the embedded document to be detected accordingly, which seems to me as a very interesting information.

[R C-R]

5 hours ago, Pšenda said:

n general, however, the application itself should not hold thousands/millions of active connections because it is obvious that it can grow above all limits, which then cause the system resources to overload.

An active connection is only required when actually reading from or writing to a file. Leaving a file open when it is not being accessed does not AFAIK require any additional work or load on the system, other than using a few bytes of RAM for the file descriptors, handles, or whatever else the OS might use to keep track of which files are open.

[lacerto]

(...)

[walt.farrell]

2 hours ago, Lagarto said:

When using Windows Task Manager's Resource Monitor, it can be seen that Affinity Publisher has all the imported files still open.

I've never noticed that information available in the Resource Monitor. Can you help me find it?

[Pšenda]

1 hour ago, walt.farrell said:

I've never noticed that information available in the Resource Monitor. Can you help me find it?

Maybe this:

But this is basic information about disk operation (transactions).

For a detailed overview of disk operations you need to use better tools, eg ProcessMonitor from SysInternals (see my previous post).

[lacerto]

(...)

[lacerto]

(...)

 

[Jerry Jenkins]

Lagarto,  thank you. You  encountered exactly the same error messages that I did; it is useful to know they are reproducible and not local to my system. also, as you, I encountered no problems running Indesign on files with over 2,000 links, so did not think it could be purely an operating system issue.  <y problems with Publisher were on an older Mac OS -- 10.10 -- Have just installed Publisher on a Mac running Mohave 10.14.3, and tomorrow will run the same files and test. Hope it works! Will post results.

J.

 

 

 

[lacerto]

(...)

[Jerry Jenkins]

Lagarto, Psenda, Walt, and all: I set up Publisher on a newer machine here, iMac Retina, Mohave 10.14.3, i7 processor, 32 GB memory, and asked it to link to several hundred Illustrator drawings of tree buds. These are fairly complex drawings, 5-10 MB each, lots of shapes and gradients. Publisher handled them without and hesitation or fuss, and saved and reopened the file without incident, in a matter of a few seconds.

So, unless I run into problems in even larger files, everything is working. My conclusion is that while Publisher runs under Mac OS 10.10, it cannot link to large numbers of vector files, either in Designer or Illustrator format, without generating errors and crashing. I love working in Publisher, and am pleased with this outcome. And grateful for your help. I will continue to work in it -- book of about 500 pages, maybe 2,000 illustrations and maps -- and am sure I will have other questions as I go along. Jerry Jenkins  norternforestatlas.org

 

 

 

 

 

 

 

 

 

[wonderings]

Out of curiosity has anyone tried this with jpegs or similar files? Same result or will it place those files in no problem in large numbers as they are not exactly editable files like a vector file would be. Or would it be like 2,000 photoshop files open and the format does not matter as Publisher is not linking files like Indesign does?

[lacerto]

(...)

[R C-R]

3 minutes ago, Lagarto said:

And the same is true with current Windows version, which is just unbearable. Publisher needs to either process files so that they are not unnecessarily left open, or it needs to use native file handling that does not have any arbitrary limit on the number of files that can be simultaneously open.

Just to clarify things a tiny bit, older versions of the Mac OS had lower default limits on the number of file references they could track compared to newer versions. But they all leave the files open until they are no longer needed by any process.

[lacerto]

(...)

[R C-R]

3 hours ago, Lagarto said:

These kinds of limitations should be explicitly stated in app specs ...

The limits I was referring to are the defaults imposed by different versions of OS X & the macOS. It is possible to change them using certain Terminal commands, but in general it is not a good idea to do that.