I’ve regarded round quite a bit to attempt to discover the reply to this however have up to now been unsuccessful. I’m programming in Python, and I’ve code that takes a very long time to run (hours to months relying on the applying) and I’m attempting to optimize it. On bigger thread techniques (as soon as I get above ≈8 threads, however particularly >18 threads), I run right into a scenario the place the system is utilizing up a considerable amount of my CPU as an alternative of the particular code I wish to run. On a 12-thread machine as I kind this, the system is utilizing ≈25–30% of my complete CPU. If I attempt to run it on a 36-thread machine, the system takes >70% of the CPU which is just unworkable (although working two cases of the code, limiting every to 18 threads, cuts this overhead again, oddly sufficient).
The Exercise Monitor signifies that taskgated is utilizing at the least 10%, whereas notifyd, logd, and launchd are additionally utilizing a number of p.c, every (together with mds at 1% so I ought to flip off Highlight, and generally lsd spikes to >40%, however that is extra uncommon … be aware that that course of is one other launch system daemon).
My older speculation was that it was a disk I/O challenge, for the reason that code was writing and studying many small information to attempt to preserve observe of sure issues and recuperate if I wanted to cease it.
My now-working speculation, based mostly on what I might discover on-line in regards to the interactions between taskgated and launchd, is that this explicit chunk of code is spawning a lot of processes and macOS’s launching and safety daemons are taking giant quantities of CPU to ensure these processes are protected. These are issues like calling “mv” and “rm” from the command line (os.system(…) in my Python code), and spawning different code that take a second or two to run (corresponding to in a distinct conda setting when two have conflicting installs of mandatory parts). I depend at the least 40 potential spots the place this chunk of code is probably spawning baby processes, and I thread it in order that it is doing that concurrently nonetheless many threads there are (so, on a 12-thread machine, 12×40 over the course of ~10 seconds). It does not appear to me like this must be making my system take up a lot CPU, however that is my finest guess in the meanwhile.
Presumably associated, the taskgated is continually spitting out to log information, “MacOS error: -67062”, which once more I’ve looked for and located no luck with diagnosing the problem. And, diskarbitrationd is spawning numerous “<non-public>” messages within the Console, however its CPU% is round 0.3% so I am much less involved about that.
My apologies for rambling a bit right here, however I am attempting to offer the data I’ve, and hopefully somebody right here has an concept. If I can eliminate this 25% or bigger challenge, that may save months of time.
For what it is value, I am working macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop computer. Identical challenge for all. Operating in Linux on an almost equivalent 36-thread system construct doesn’t have the overhead challenge (however I actually do not wish to swap to Linux) which is another excuse why I do not suppose it is a disk I/O downside.
