Why download the addons seperately for photo and publisher seperately?

[GoaTrance]

Just wondering why there is a lack of co-ordination with the products. I just downloaded 2Gb of addons for product(s) when it could have been half that if the products knew that it was already done in a common place. Same machine, same drive. I don't think that's really very good and could have taken a bit more thought about the distribution.

I'm about saving data, preventing malware and all sorts of internet issues, I'm sorry to say duplicate downloads for no reason are a cause for concern. That's a big download for no discernable reason bearing in mind the link between the products.

Every little bit in saving resources counts. This is not helpful.

Kindest Regards.

 

[edwardsson]

+1

[sfriedberg]

And the problem is amplified when Serif doesn't give us the ability to control which drive/directory/folder is used.  I want to control where stuff is stored on my machine.  Downloading three copies of stuff and having it all stashed on the (deliberately!) limited C drive is an unnecessary, and unpleasant, experience.

[Pixelkobold]

Same problem here. 
I find the new feature awesome, but I don't see why it is necessary to download and install all the addons from serif shop into all three different apps seperatly and so triple,
especially since some of them are really huge and need diskspace on the systemdrive which I dont have to spare.

I agree with sfriedberg. The possibility to give at least a specific folder for the addons (on antoher harddrive) would be great. 
I see that it could be difficult for brushes, but I think for assets and such it wouldnt be a great problem to have that function

 

anyway serif,

you do a great job, I live your apps. Keep on like this

[R C-R]

14 hours ago, GoaTrance said:

Just wondering why there is a lack of co-ordination with the products.

From what the staff have said in the past, the biggest & hardest to resolve issue is the sandboxing restrictions recent versions of both the Mac & Windows OS put on low level cross-application data sharing. They could probably kludge up something to work around that but I suspect it would be difficult to maintain the stability & performance of that going forward.

[walt.farrell]

27 minutes ago, R C-R said:

From what the staff have said in the past, the biggest & hardest to resolve issue is the sandboxing restrictions recent versions of both the Mac & Windows OS put on low level cross-application data sharing. They could probably kludge up something to work around that but I suspect it would be difficult to maintain the stability & performance of that going forward.

These days they may also face issues with various malware-protection products (especially ones protecting against ransomware) which do a similar kind of sandboxing and prevent one application from writing into files that are part of another application.

[Pixelkobold]

 

21 hours ago, R C-R said:

From what the staff have said in the past, the biggest & hardest to resolve issue is the sandboxing restrictions recent versions of both the Mac & Windows OS put on low level cross-application data sharing. They could probably kludge up something to work around that but I suspect it would be difficult to maintain the stability & performance of that going forward.

didn't thought about that, but that does make sense somehow

[loukash]

On 2/6/2021 at 7:56 PM, R C-R said:

the biggest & hardest to resolve issue is the sandboxing restrictions recent versions of both the Mac & Windows OS put on low level cross-application data sharing. They could probably kludge up something to work around that but I suspect it would be difficult to maintain the stability & performance of that going forward.

Um, that's not "recent versions". Sandboxing was here since Affinity shipped in 2014, and it was to be expected that it just becomes even more restrictive with each MacOS upgrade.

But it's definitely one of the reasons I'll be buying the v2 upgrades of APh and AD from the Serif store and not from Mac App Store.
Whereas I bought APu 1.7 from the Serif Store right from the start, and can thus do with the ~/Library/Application Support/Affinity Publisher folder whatever I want, like moving it into any location of my choice by means of simple symbolic link magic (which is exactly what I did and I have my reasons for doing it). Not possible with App Store sandboxed apps and their walled garden inside ~/Library/Containers that prevents symlinks from "looking outside".

But the biggest conceptual mistakes Serif made were:

• using completely separated app support folders for each app, instead of organizing them into subfolders:
  ~/Library/Application Support/Affinity/Designer
  ~/Library/Application Support/Affinity/Photo
  ~/Library/Application Support/Affinity/Publisher
  and adding:
  ~/Library/Application Support/Affinity/Shared
• putting all assets into a monolithic file called assets.propcol – my Photo/Designer 1.9*) assets.propcol are 2.58 GB each! WTF?! In fact, they both contain the same 1+ GB freebie which Serif gave away a few years back and that I have already stored elsewhere on the drive!

As for shared data in a sandbox: isn't it what ~/Library/Group Containers is for? Serif has a few folders in there aready:
~/Library/Group Containers/6LVTQB9699.com.seriflabs
~/Library/Group Containers/6LVTQB9699.com.seriflabs.beta
~/Library/Group Containers/6LVTQB9699.com.seriflabs.extensions

~~~

*) While I've downgraded to 1.8.4 on my everyday's El Capitan, today I have installed the full v1.9 suite on a spare High Sierra partition for experimenting.

[walt.farrell]

5 minutes ago, loukash said:

Um, that's not "recent versions". Sandboxing was here since Affinity shipped in 2014, and it was to be expected that it just becomes even more restrictive with each MacOS upgrade.

And they must also worry about however the Windows Store does sandboxing.

[loukash]

For what it's worth though, Apple is only restrictive with sandboxed user content. At least up to Catalina – no idea about the newest Big Sur restrictions, lacking a compatible Mac.
But stuff outside a sandbox can be easily "outsourced" via symlinks. Most standard apps will have no problems with that:

My Catalina partion is only 64 GB in size. Yet it can run full Xcode, Logic Pro with its complete sound library, all Affinity apps, iMovie, GarageBand, Pages, Numbers, you name it, all at the same time (not that I'd do that, but hey, it's fun, right?). After I have downloaded and installed an app, I moved it to another large partition on this SSD and linked it back to the Catalina /Applications or /Library folders via symlink. (Logic has this functionality already built in, for relocating its 70+ GB sound library to a different volume). So I have a 64 GB partition that runs around 150 GB worth of apps and their supporting files.
(Dads, don't try that at home on your kid's brand new shiny iMac! )

That would easily work with non-sandboxed Affinity user files if only the apps would be able to share them. (Note to self: check out if the non-sandboxed APu can share some *.propcols from ~/Library/Containers/com.seriflabs.affinitydesigner/Data/Library/Application Support/user)

[walt.farrell]

1 hour ago, loukash said:

(Note to self: check out if the non-sandboxed APu can share some *.propcols from ~/Library/Containers/com.seriflabs.affinitydesigner/Data/Library/Application Support/user)

Experimentation has shown (painfully) that the format of some .propcol files is different between the apps, at least on Windows. Trying to replace an assets.propcol in Designer with one from Photo (or possibly vice-versa) causes application failures, for example.

But that's copying from one application to another.

Sharing has some different issues, potentially, because the application does not expect that anyone else might be writing to the .propcol file at the same time it is writing (or reading). You could end up with application failures or data loss in several scenarios of simultaneous access, or if the OS only allows one application to have the file open.

Keep careful backups if you play with this

[loukash]

35 minutes ago, walt.farrell said:

format of some .propcol files is different between the apps, at least on Windows

It's very likely the same format on both platforms, so thanks for the words of caution.

37 minutes ago, walt.farrell said:

Sharing has some different issues, potentially, because the application does not expect that anyone else might be writing to the .propcol file at the same time it is writing (or reading).

Good point.

37 minutes ago, walt.farrell said:

You could end up with application failures or data loss in several scenarios of simultaneous access, or if the OS only allows one application to have the file open.
Keep careful backups if you play with this

No worries.
As noted:

3 hours ago, loukash said:

I have installed the full v1.9 suite on a spare High Sierra partition for experimenting

I.e. I can reformat and reinstall the complete partition at any given time. Nothing here that's not replaceable, all is safe.

Just a few minutes ago I've finished here some "dirty testing" of Nik Collection 2015 within Photo 1.9:

… ff.

[v_kyr]

50 minutes ago, walt.farrell said:

... Sharing has some different issues, potentially, because the application does not expect that anyone else might be writing to the .propcol file at the same time it is writing (or reading).

Possible concurrent write access can be a problem here, if not handled via explicite locking and releasing of a shared resource. Concurrent read access is in contrast to write access usually not that problematic, though can also be done safely with a lock/release mechanism.

[walt.farrell]

3 minutes ago, v_kyr said:

Possible concurrent write access can be a problem here, if not handled via explicite locking and releasing of a shared resource. Concurrent read access is in contrast to write access usually not that problematic, though can also be done safely with a lock/release mechanism.

Right, but writing with a concurrent read can also have some serious problems, including application failure or data loss. Especially if the application doing the read doen't fail, and later writes the (incomplete or corrupt) data back out to the file.

[Frozen Death Knight]

On 2/6/2021 at 9:06 AM, sfriedberg said:

And the problem is amplified when Serif doesn't give us the ability to control which drive/directory/folder is used.  I want to control where stuff is stored on my machine.  Downloading three copies of stuff and having it all stashed on the (deliberately!) limited C drive is an unnecessary, and unpleasant, experience.

Fully agreed. I do hope that these issues will be a high priority fix for 1.10 now when they have a dedicated team specifically made for dealing with the new account downloading system. This new account feature is a nice convenience, but it most certainly has a lot of room to grow.

The devs at least seemed interested in this feedback during the Beta period, so let's see if they are able to deliver for the next major update.

[walt.farrell]

On 2/6/2021 at 3:06 AM, sfriedberg said:

And the problem is amplified when Serif doesn't give us the ability to control which drive/directory/folder is used.  I want to control where stuff is stored on my machine.  Downloading three copies of stuff and having it all stashed on the (deliberately!) limited C drive is an unnecessary, and unpleasant, experience.

Potentially 6 copies, if you run the betas, too.

We have a kind of model for user-specified locations already, in that the Preferences dialog lets the user specify where the user has installed additional dictionaries. Or the specification of locations for templates in the File > New dialog.

But by the time the user can open Preferences to specify a directory location, Affinity has already created all the user's settings files (including assets.propcol, raster-brushes.propcol, etc. containing the standard content. So there would need to be some kind of migration mechanism associated with that Preference setting so that Affinity could move the data from the original default location to the user's newly specified location.

Of course, we (and the developers) also know that having a massive, monolithic assets.propcol file is not really a viable long-term design. Especially with some of the huge asset packs that are available from the Affinity Store. That's a recipe for disaster if there are any issues with reading/writing the file. So there's already some significant redevelopment work needed in the content management area.

[Patrick Connor]

This is no different than the situation before this feature was introduced, (with the exception of not having another copy in your downloads folder) all resources were manually installed into each individual application making a copy as it was imported.

It will need carefully consideration to change to shared resources.

[walt.farrell]

5 minutes ago, Patrick Connor said:

It will need carefully consideration to change to shared resources.

Yes, it will.

But note that that's only one of the considerations being mentioned here. Another is allowing the user to specify the location for storing the content. Ideally both would be provided. Though sharing would reduce the total space required, even one copy may be more than the user wants to have on their C drive.

 

[v_kyr]

47 minutes ago, walt.farrell said:

Right, but writing with a concurrent read can also have some serious problems, including application failure or data loss. Especially if the application doing the read doen't fail, and later writes the (incomplete or corrupt) data back out to the file.

Most filesystems (but not all) use a locking to guard concurrent access to the same file. The lock can be exclusive, so the app to get the lock gets access - subsequent apps get a "access denied" error. So for example APh will be able to read the file and gets the file lock, but ADe will not be able to write since APh is reading. Some filesystems (for example NTFS) allow the level of locking to be specified, to allow for example concurrent readers, but no writers. Byte-range locks are also possible.

However, unlike databases, filesystems typically are not transactional, not atomic and changes from different apps are not isolated (if changes can even be seen - locking may prohibit this). Using whole-file locks is a coarse grained approach, but it will guard against inconsistent change/updates. In the past not all filesystems supported whole-file locks, and so instead it's common practice then to use a lock file - a typically empty file whose presence indicates that its associated file is in use. Creating a file is an atomic operation on most file systems. - See also file locking which gives a general overview.

[R C-R]

52 minutes ago, loukash said:

... putting all assets into a monolithic file called assets.propcol ...

As I understand it, each of the *.propcol files is binary, optimized in some unspecified ways for performance (& perhaps for memory efficiency), not for minimal file size. So in addition to the other considerations regarding sharing/locking/syncing & such, that has to be a part of it as well. For example, with a single file there may be no need to open a bunch of different ones for different assets if it is organized sort of like a database or with identifiers & pointers so chunks of data can be accessed as needed.

1 hour ago, loukash said:

As for shared data in a sandbox: isn't it what ~/Library/Group Containers is for? Serif has a few folders in there aready:
~/Library/Group Containers/6LVTQB9699.com.seriflabs
~/Library/Group Containers/6LVTQB9699.com.seriflabs.beta
~/Library/Group Containers/6LVTQB9699.com.seriflabs.extensions

FWIW, 6LVTQB9699 is Serif's Apple developer ID. I have no idea of the details but I think what goes into those folders has something to do with Apple's ever tighter security model that permits only limited data sharing among apps even from "identified developers." IOW, they are not intended for general data sharing (& for many apps contain only empty folders or ones with a few hundreds of bytes of data). 

[loukash]

9 hours ago, walt.farrell said:

monolithic assets.propcol file is not really a viable long-term design. Especially with some of the huge asset packs that are available from the Affinity Store. That's a recipe for disaster if there are any issues with reading/writing the file. So there's already some significant redevelopment work needed in the content management area.

^ requoted for emphasis!

9 hours ago, Patrick Connor said:

This is no different than the situation before this feature was introduced, (with the exception of not having another copy in your downloads folder) all resources were manually installed into each individual application making a copy as it was imported.

I'm aware of that, and I found it highly annoying from the start. But back then I wanted to check out all those free-as-in-beer brushes et al, so I had to play along…

9 hours ago, Patrick Connor said:

It will need carefully consideration to change to shared resources.

Yep, that's a significant part of any devs' job.

6 hours ago, R C-R said:

As I understand it, each of the *.propcol files is binary, optimized in some unspecified ways for performance (& perhaps for memory efficiency), not for minimal file size. So in addition to the other considerations regarding sharing/locking/syncing & such, that has to be a part of it as well.

That's what ~/Library/Caches is for. Aside from the already present folders com.seriflabs.affinitydesigner, com.seriflabs.affinityphoto and com.seriflabs.affinitypublisher, an app can create its own "private" folder in there and cache whatever else it needs for fast access. If a user deletes that folder, the app will simply rebuild its cache on the next launch.

Some apps will let you choose the path where to put a cache or temporary files, which is in particular necessary for audio editing apps that usually use huge 32-bit floating point files. ~/Library/Caches has the benefit that it's excluded from Time Machine backups so you don't store needless junk on your precious backup drive over and over again.

Some other apps, on the other hand, are poorly conceived and store their cache/temp junk in ~/Library/Application Support or even in ~/Documents where it's being pointlessly backed up by Time Machine again and again – unless you'd explicitly exclude it from TM backup manually. For those apps I'd usually create a cache folder in ~/Library/Caches myself and redirect them via symlink; iZotope Product Portal – yet another cross-platform app with a non-standard UI – being a recent bad example with 200 MB of cache junk in ~/Library/Application Support/iZotope/ProductPortal_cookies that don't need to be backed up: at worst you'd have to log into your iZotope account again.

6 hours ago, R C-R said:

I have no idea of the details but I think what goes into those folders has something to do with Apple's ever tighter security model that permits only limited data sharing among apps even from "identified developers." IOW, they are not intended for general data sharing (& for many apps contain only empty folders or ones with a few hundreds of bytes of data). 

I have no exact idea either, it was just a "fraction-educated" guess. Even though I still have an Apple Developer ID from the days when membership was free, my programming skills are sadly limited to basic AppleScript.

[R C-R]

5 hours ago, loukash said:

11 hours ago, R C-R said:

As I understand it, each of the *.propcol files is binary, optimized in some unspecified ways for performance (& perhaps for memory efficiency), not for minimal file size. So in addition to the other considerations regarding sharing/locking/syncing & such, that has to be a part of it as well.

That's what ~/Library/Caches is for.

From a quick sample of what other apps store in those folders, I don't think that is what they are for. Mostly, they seem to have relatively modest sized Cache.db, Cache.db-shm, & Cache.wal files plus maybe a few others specific to various apps. From what I read on the web, the db files are intended for temporary storage only. At least on my system only APub bought from Serif puts anything there -- AP & AD bought through the MAS do not.

Anyway, the data structure of the *.propcol files seems very different from anything in ~/Library/Caches. 

[loukash]

1 minute ago, R C-R said:

modest sized Cache.db, Cache.db-shm, & Cache.wal files

Those are SQL databases handled by MacOS. But an app can store anything else in the Caches folder, if it only wants.
Example off the top of my … folder list:
~/Library/Caches/Adobe/Bridge CS5.1/Cache/1024 – the latter being 1024 px previews – plus other Bridge items in the same directory.
Or:
~/Library/Caches/Adobe InDesign/Version 7.5/de_DE with plethora of custom files and folder inside, including InDesign Recovery.
Or:
~/Library/Caches/Metadata/MacJournal which are Spotlight proxies to access MacJournal entries otherwise hidden inside its MacJournal Data6.mjdoc packaged document (which in fact consists of individual RTF and RTFD files that were otherwise unaccessible to Spotlight for indexing)

Anything's possible. It's up to the developer.

12 minutes ago, R C-R said:

AP & AD bought through the MAS do not.

Those are in ~/Library/Containers/com.seriflabs.affinitydesigner/Data/Library/Caches/com.seriflabs.affinitydesigner because sandbox.

13 minutes ago, R C-R said:

Anyway, the data structure of the *.propcol files seems very different from anything in ~/Library/Caches. 

For sure.
The idea is that Affinity would be loading the content of my individually and centrally stored assets/brushes/whatever from my own folder that I would have defined in Affinity Preferences, and create an optimized cache of it in ~/Library/Caches for quick access. If the cache gets purged, it would be simply recreated on the next app launch.

That's e.g. what NeoFinder (previously CDFinder) does since decades:
By default, the folder where NeoFinder stores its catalogs is in ~/Library/Application Support/NeoFinder/NeoFinder Database
But in Preferences, you could always change the location of this database folder to wherever you want. E.g. into Dropbox for easy sync between devices.
Older versions would then store a monolithic "de.wfs-apps.neofinder.quicklaunch.cache" file in ~/Library/Caches, the current version now uses a different approach with subfolders.

[R C-R]

19 minutes ago, loukash said:

Those are SQL databases handled by MacOS. But an app can store anything else in the Caches folder, if it only wants.

But the issue is to what extent the macOS in all its supported versions permits other apps (even those from the same developer!) to share the data in those caches folders, if at all. In essence, you are talking about defeating sandboxing. That has serious privacy & security implications, no matter how carefully it is done.

26 minutes ago, loukash said:

The idea is that Affinity would be loading the content of my individually and centrally stored assets/brushes/whatever from my own folder that I would have defined in Affinity Preferences, and create an optimized cache of it in ~/Library/Caches for quick access. If the cache gets purged, it would be simply recreated on the next app launch.

There is no guarantee that a bunch of individual property files would provide quicker access than a single binary collection file would, particularly if 3 different apps are trying to share the same ones through various launches, memory purges, & so on. Allowing any of that to be stored outside of the local user's home folder, particularly in a remote location like Dropbox, adds a whole new layer of complication that almost certainly would adversely affect performance, as well as the overall security of the system.

Consider also that this must work regardless of which store the apps are bought from, which means Serif must abide by the sandboxing restrictions imposed by Apple & Microsoft on apps bought through their stores, as well as any imposed by any version of the OS the apps support. 

[loukash]

15 minutes ago, R C-R said:

the issue is to what extent the macOS in all its supported versions permits other apps (even those from the same developer!) to share the data in those caches folders, if at all. In essence, you are talking about defeating sandboxing. That has serious privacy & security implications, no matter how carefully it is done.

Yeah, I guess it wouldn't work for the App Store versions.

15 minutes ago, R C-R said:

a bunch of individual property files

I'm not saying that. Affinity could store its "assets.propcol-as-a-cache" wherever they want. It's just that it should be handled as that: a cache.

15 minutes ago, R C-R said:

the overall security of the system

Haha, that's why they will have to pry my 2012 MacBook from my cold, dead hands. I'm old enough to handle all consequences how I mess up the OS of my computer. Big Sur and beyond? Thanks, but no, thanks. The walls of that garden are too high for my taste. (Your mileage may vary.)

Still, there are possibilities: It's not a coincidence that many developers actually left the App Store due to overcautious sandbox requirements.

I'm only buying on App Store if there's no other way, as was the case with the initial discounted versions of APh and ADe in 2014 and 2015.

15 minutes ago, R C-R said:

Consider also that this must work regardless of which store the apps are bought from, which means Serif must abide by the sandboxing restrictions imposed by Apple & Microsoft on apps bought through their stores, as well as any imposed by any version of the OS the apps support. 

Of course. Hey, we're just brainstorming here. One can still dream, right?