Too many open files message

[Jerry Jenkins]

Hi,

Using Publisher on a Mc, OS 10,10.5. Working on a book chapter with many small linked Illustrator files. When I got to about 150 linked files I started getting error messages in a a pop-up box that says "Failed to open file" and gives me the name of one of the linked files, and then "The file could not be opened because there are too many open files in the system." After I click through a hundred or so of these boxes, I can get access to the file, and I see that many of the linked files are marked "missing" In resource manager.  The files are where they have always been; Publisher is not linking to them. If I attempt to work on the file, I get the whole string of a hundred popups again.

I work routinely on illustrated books with 1,000 or more links, so need a work-round for this.

Questions for the Forum:  Has anyone else seen this? Is this a bug, or is there a known limit to the number of linked files you can have? Any suggestions? Can I get around this by embedding them?

As a trial I am recreating the file with all embedded images. I will report on what happens.

Thank you,

J.

 

 

 

[Pšenda]

On 3/20/2020 at 3:51 PM, Jerry Jenkins said:

Has anyone else seen this?

Try here:

https://www.google.com/search?q=Too+many+open+files+message+site%3Aforum.affinity.serif.com

There are a few posts with the same problem, maybe they will help you.

[Jerry Jenkins]

Psenda, 

Thank you, and I did look at those. One seems similar, but dealt with PSD files. I was wondering in what situations others have encountered this, and if anything was known about wht the limits to open files were.

 

J.

[walt.farrell]

If you purchased via the Mac App Store, I think that MacOS might enforce a limit on the number of files that Publisher (or any other MAS application) can have open.

If you have a lot of fonts installed, I think that will also contribute to the issue. Possibly you could use a font manager to reduce the number of fonts to just the ones you're actively using, which would allow Publisher to open more of your illustration files.

(Disclaimer: Not a Mac user; just making some inferences from things I've read in these forums.)

[Gear maker]

9 minutes ago, walt.farrell said:

If you purchased via the Mac App Store, I think that MacOS might enforce a limit on the number of files that Publisher (or any other MAS application) can have open.

I hope we don't ever run into an issue that where it is purchased makes a difference in it's capabilities.  I can understand it if it's a platform change, but not where it's bought.  "I'm sorry but you'll have to buy another copy this time from the abc store to be able to do that" would not be a good answer to get.  If that were the case then it shouldn't be sold from wherever limitations are imposed.

[walt.farrell]

4 minutes ago, Gear maker said:

I hope we don't ever run into an issue that where it is purchased makes a difference in it's capabilities. 

We already have, with the Microsoft Store, and (as I mentioned above) I believe with the Mac App Store.

Both stores seem to apply various sandboxing techniques to improve system and application security. And application developers don't necessarily know what those techniques are nor what they will affect. Nor how they may change from release to release of the OS.

[Gear maker]

2 minutes ago, walt.farrell said:

We already have, with the Microsoft Store, and (as I mentioned above) I believe with the Mac App Store.

I may never buy another application from the Mac Apple Store!  When I bought AD and APh that was the only choice.

[wonderings]

I wonder if it is because those linked files are viewed as editable elements and not placed/linked files like you would have in Indesign. I wonder if Illustrator would have a similar limit if you tried to open 150 .ai files at once. Could this be another issue stemming from not allowing you to simply place an image/pdf/vector without the need to edit it?

[wonderings]

30 minutes ago, walt.farrell said:

If you purchased via the Mac App Store, I think that MacOS might enforce a limit on the number of files that Publisher (or any other MAS application) can have open.

If you have a lot of fonts installed, I think that will also contribute to the issue. Possibly you could use a font manager to reduce the number of fonts to just the ones you're actively using, which would allow Publisher to open more of your illustration files.

(Disclaimer: Not a Mac user; just making some inferences from things I've read in these forums.)

Maybe I am reading the OP question wrong, but this is not 150 publisher files but 150 placed files in Publisher. 

[Pšenda]

21 minutes ago, wonderings said:

but this is not 150 publisher files but 150 placed files in Publisher. 

Every open file has Handle (I am Windows user), which is assigned from the OS when the file is opened. Then it does not matter whether the files are open directly by the application or the linked files are open to display the contents of one source file.

[Jerry Jenkins]

Thank you all, and this is proving interesting.

To clarify. Publisher was bought from Affinity, not the Mac store. The problem is with AI files placed as linked files on Publisher pages. The error message "Failed to open file ... too many open files in system" comes from Publisher, but it may still reflect an operating system problem.

It is not, however, a general problem; in Indesign I routinely work on documents with 1,500 to 2,000 linked files, without problems. But it may be, and I am just speculating here, that Publisher keeps all the placed files open, and Ind doesn't.

It is, however, a dangerous problem: once it has occurred, when you try to save you get the message "Failed to save because access to the file was lost," so the file is no longer usable. Were this to happen several months into a big project, it would be very bad. 

I am currently trying to workarounds: recreating the test file with all the placed files embedded; and converting the placed files to Designer files, incase the problem has to do with the way Publisher handles illustrator files. I will take your comments into account as I do this, and report what I find.

With thanks,

J.

 

 

 

 

 

[walt.farrell]

13 minutes ago, Jerry Jenkins said:

But it may be, and I am just speculating here, that Publisher keeps all the placed files open,

It has to keep them open, as otherwise it won't know if they've been changed but another application.

Thanks for clarifying that it's not from the Mac Store; that removes one variable

How many fonts installed?

Also, as Mac is a Unix-style OS, you can probably changes the default file handle limit for your user, or system-wide. I don't recall off-hand how to do that. Sorry.

[Pšenda]

1 hour ago, walt.farrell said:

It has to keep them open, as otherwise it won't know if they've been changed but another application.

It is not necessary to hold the file open to detect changes in the contents of the file. Changing a file is detected by other means, as is done when the source file is opened when it is determined that the linked files have been changed since the previous state.

[Jerry Jenkins]

Hello all,

In answer to Walt, two type faces, each used in regular, italic, bold, small caps. So likely 8 font files.

Just finished and two experiments. First made a test file and embedded about 150 AI files in it. It saved and open fine, but of course the file size got big fast. The Publisher help file says that there is a file size beyond which it won't allow embedding, but doesn't say what that is.  I am starting a 500-page book with 10-15 vector drawings per page, so embedding all the drawings is not an option, unless I break it into smaller files.

Second experiment, I added linked AI files in batches of thirty to a test file. Up to about 130 files everything was fast and fine.The next batch of 30 generated error messages and crashed the save.

Tomorrow I will experiment with adding embedded and link tiff files to see if the problem is files in general or specific to Illustrator files. I will post the results.

With thanks for your help. I am really hoping I solve this because I love working on Publisher. But my work demands that I link to thousands of files, and it may be that the current build of Publisher is not ready for that.

J.

 

 

 

 

 

 

 

 

 

 

[Move Along People]

-

[Pšenda]

15 minutes ago, Jerry Jenkins said:

but of course the file size got big fast

What version of APu do you use?

Just interesting - why do you have so many enters in each post? It's hard to read.

[Move Along People]

-

[walt.farrell]

32 minutes ago, Jerry Jenkins said:

In answer to Walt, two type faces, each used in regular, italic, bold, small caps. So likely 8 font files.

Not "how many fonts are you using?", but "how many fonts are installed?" Affinity enumerates (and opens, perhaps?) all the fonts that are installed, and I think they count toward your allowed maximum number of open files.

[walt.farrell]

3 hours ago, Pšenda said:

Changing a file is detected by other means, as is done when the source file is opened when it is determined that the linked files have been changed since the previous state.

But Publisher will also detect those changes in real-time, and warn you if any Linked file is changed, instantly as it is changed. (And, optionally, update your view in Publisher instantly.)

This is different from detecting changes as the Publisher file is opened. And I believe it requires maintaining a handle to each Linked file.

[Jerry Jenkins]

All,

Two more experiments. I made a test file, without text, and placed about 200 linked TIFFS in it. Publisher behaved sweetly, fast, no problems. Saved quickly, reloaded quickly.

I converted 160 Illustrator files to Affinity Designer files,  made another test file, placed them as linked files. Everything worked fine until I hit about 140 files, at which point Publisher freezes when I try to save.

So the problem is specific to vector files and occurs at around 150 files per document. And would seem to be a limitation of the either my hardware (Mac Retina, 4HZ i7 16 GB ram) or of Publisher.

It may be the deal breaker for me. Should I report this as a possible bug?

Once again, many thanks for your help.

 

J.

 

 

 

  

 

[R C-R]

12 hours ago, Jerry Jenkins said:

Using Publisher on a Mc, OS 10,10.5.

10.10.5 is the last update for OS X 10.10 Yosemite, now almost 5 years old. Is there some specific reason you have not upgraded to a newer macOS version than that?

I ask because it is the OS that sets the limit on the number of files that can be open at the same time. (Technically, it is the number of file descriptors that can be referenced by the system, but it is nearly the same thing.) There are also two limits, one for the number any one process can have open (reference) & another for the total for all processes. These limits are there to improve performance & to protect the system from running out of resources to track open files.

Because many resources (like fonts) are shared by different processes & processes are structured hierarchically, there is no simple way to determine how many files are being tracked because any one application needs access to them. (Monitoring Activity Monitor in its "All Processes, Hierarchically" view mode may give you some idea of how it works.)

If you search the web on something like "What is the limit on number of files open in MacOS?" you can find a lot of info about about how to change the limits, but it is neither simple, without dangers, nor the same for different Mac OS versions.

Anyway, I am not at all sure about this but I think more recent versions of the macOS handle the limits better than older ones, so the safest & easiest solution may be to upgrade your Mac's OS to a newer one unless there is some compelling reason to avoid doing that.

[Pšenda]

9 hours ago, haakoo said:

and that's why it skips a line.

Unfortunately, for me shows these redundant Enters, and it is difficult to read - the posts are broken with this.

@Jerry Jenkins could these redundant Enters be removed? Posts can be edited. Thank you.

[Pšenda]

9 hours ago, walt.farrell said:

But Publisher will also detect those changes in real-time, and warn you if any Linked file is changed, instantly as it is changed.

Yes, but it is not necessary to keep these files "open" for realtime monitoring of file changes, or for realtime showing this change.
Maintaining an active link to a file (opening it for reading) is of course very burdensome for the OS, so it is obvious that it will not be desirable for thousands of files at the same time.
Take a OneDrive/Dropbox that maintains "realtime" synchronization between thousands of your files and the cloud. Do you think he has all these files active open?

[walt.farrell]

1 hour ago, Pšenda said:

Take a OneDrive/Dropbox that maintains "realtime" synchronization between thousands of your files and the cloud. Do you think he has all these files active open?

No, but both of those operate at a deeper level in the system, as drivers, rather than as normal applications. The Affinity applications operate at the normal application level, and I don't know what facilities Mac or Windows provide to a normal application that allow monitoring for file changes without having the file open.

[R C-R]

59 minutes ago, Pšenda said:

Maintaining an active link to a file (opening it for reading) is of course very burdensome for the OS, so it is obvious that it will not be desirable for thousands of files at the same time.

I don't know how it works in Windows but in the Mac OS there is really nothing that could be considered particularly burdensome about keeping files open for reading, as long as there is enough available RAM for all of their file descriptors. The descriptors are just reference numbers, so tracking even thousands of them uses very little RAM.

I run macOS Mojave (10.14.6). I used a Terminal command to check its open file limits. The total limit returns "unlimited" so that is not a problem. The per process limit is 256, which sounds low, but since apps rely on many different processes & many of them are 'owned' by some other process, it is unlikely that would be a problem either.