Plex, rclone mounts and bandwidth usage
I've been having some fun lately, setting up Plex on a new VPS, mounting my music share using rclone mount.
I've been using the same setup previously, on a slower node, but with only outbound bandwidth measured. There I mounted my music from pCloud.
On this new node I figured I would try out/use @koofr. So I mounted the drive, using
/usr/bin/rclone mount --uid 1000 --gid 1000 --syslog --stats 1m \ -v --allow-other --read-only myKoofr:/music/ /srv/music/
I installed plexmediaserver and added 149 GB of music to the library, from that Koofr-rclone mount.
That triggered Plex to download ~900 GB of data ...
(Actually jumped in at ~600 and tried figuring out what's going on. By then it was halvway through indexing/scanning the library. @AnthonySmith suggested I could try to add --ignore-checksum, so I did. This might have helped, as it "only" used 300 GB for scanning the last 150 GB.
I also tried pCloud, and I tried adding some more options for caching, using something like this:
/usr/bin/rclone mount --uid 1000 --gid 1000 --syslog --stats 1m \ --buffer-size 1G --no-modtime --dir-cache-time 90m \ --cache-dir /tmp/rclone-cache --vfs-cache-mode full \ --vfs-cache-max-size 1G \ -v --allow-other --read-only --ignore-checksum \ my-pcloud:/music/ /srv/music/
Still a rescan of my library used 150 GB of traffic. Maybe --no-modtime prevents it from seeing ctime/mtime, so it has to re-download every file to check ... D'oh! 
Trying again, a Scan Library Files after removing that option, with only 3 albums (re)added, it seems that still triggered a full re-download ... (Actually waiting for it to finish, but looks like it's downloading a lot.) I would think Plex stored in it's db the timestamps etc, so it wouldn't have to redownload everything every time. 
I've seen recommendations on using sshfs instead, but this makes me curious ... I could of course try davs/dav2s (the cloud providers don't offer ssh, I think). But users like @Mason and @Wolveix do run rclone (albeit with GSuite, IIRC), so it should be feasible.
If nightly rescan triggers redownload of every file, it's really not very nice ... Not a vital service for me, of course, but it triggers my curiosity. 
Maybe I'm missing something obvious, or maybe you have some insight to share on this?
Comments
Ok it's technical
Wasn't sure if it was mostly technical or mostly off-topic
Aye, I use GSuite mounted via rclone. I agree doing it this way does generate a lot of bandwidth/traffic (mostly incoming, little outgoing except when serving content).
Some tweaks I'd suggest, you can increase your "--dir-cache-time" flag in your second mount command. I personally use 72h. Will keep things hanging around in cache longer, but also detects changes in the mount for newly added files so don't worry about it not picking up new additions.
Second, check your PMS' "Library" settings. You can disable a lot of the automatic scans so that scans aren't kicking off so frequently (see below). You can up the library scan interval to the max ("Daily") to further limit the number of re-scans.
Third, check your PMS' "Scheduled Tasks" settings. Untick the 'Refresh local metadata every three days', 'Update all libraries during maintenance', 'Refresh music library metadata periodically', and 'Perform extensive media analysis during maintenance' settings.
Closing thoughts -- If you do all the above, then scans should only occur during the daily re-scan and not when files get added/changed (or when you trigger a manual scan via Library Name (in sidebar) -> three dots -> "Scan Library Files"). Your use case does seem really high in the amount of bandwidth that it uses, which may be a side effect of koofr/pCloud. I know for GDrive, it'll download the file in chucks which (to Google) translates to the file being downloaded multiple times. And depending on the media type, the whole file might need to be downloaded in order to do media/metadata analysis on the file (I don't use Plex for music so I can't comment there). I wouldn't recommend using sshfs, it'll way underperform compared to rclone w/ vfs caching. For GSuite in particular, sshfs is way more likely to cause temporary bans due to the high number of API calls (not sure how other providers handle this).
Humble janitor of LES
Proud papa of YABS
Would sshfs not stop the entire download though in a bandwidth-constrained environment?
I recommended that as that was my thinking, assuming performance does not degrade too much that is.
I need to go through a similar exercise myself, I was thinking of testing sshfs first failing that, iscsi, I believe the key is to get the head to believe the data is local, which is never will with rclone etc.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
My opinions are mainly based on using Plex for video streams, so it may perform quite well for smaller (i.e. music) files. So I guess take my advice with a grain of salt if you're only using Plex for music. When I used a FUSE-based mount, I had a lot of issues with the stream just taking forever to finally start, skipping didn't work well, and the API issues I mentioned due to not caching the content. Granted, I've been married to rclone for a good two or three years now, so my knowledge on this is quite outdated.
Would be interested in how an experiment goes under these various conditions.
Humble janitor of LES
Proud papa of YABS
Same here, usually 1 - 2 GB Movies.
I will give it a test and see how it goes.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
Newer Plex versions scan your TV Shows for intros and show a button to skip opening credits. You can disable that too. The option should be buried somewhere in the preferences.
Can confirm this. I would get banned everyday before I started using rclone.
My experience adding about 1500 music album, I use small vfs-read-chunk-size 8MB, 128MB for vfs-read-cache-limit and use 16MB for buffer-size for rclone Gdrive mount.
Also, I added 20 to 30 album per-scan to make sure Plex scan effective. I would avoid automatic & periodic scan and only ticked "Run partial scan when changes are detected". If you have high RAM VPS/server you could increase the amount of album per scan.
Complete scan of 280 GB flac library take around 1000 GB of bandwidth.
This is my mount config on 1GB VPS using rclone v.1.52.3
/usr/bin/rclone mount \ --rc \ --allow-other \ --buffer-size 16M \ --multi-thread-streams 0 \ --dir-cache-time 48h \ --tpslimit 10 \ --vfs-read-chunk-size 8M \ --vfs-read-chunk-size-limit 128M \ -v \ --async-read=false \⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.
First of all: Thanks for great replies!
Thanks! I'm thinking nightly scans will not be feasible if it's going to use 150 GB per night for files that doesn't change much.
Looking into @chocolateshirt's suggestions on vfs-chunk-size (was already considering those).
I wonder why it reads the files multiple times ... AirSonic and JellyFin used ~152 GB of traffic to scan ~150 GB of music ...
Trying your settings. Trigger to trigger partial scan, how to make Plex see the change ... Changed the name of a directory, didn't seem to help. I see you don't use --read-only(?), make that makes a difference.
What does
--rcdo? I don't see it in the web page onrclone-mount...(Edit: Found it, commented below.)
To be honest, this thread to me is looking more like an advert to have your media (whatever it is) on a local array if that is at all possible.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
try the following flags instead
--allow-other \
--copy-links \
--no-gzip-encoding \
--no-check-certificate \
--allow-non-empty \
--umask 000 \
--vfs-cache-mode writes \
--no-modtime \
--max-read-ahead 100M \
--drive-chunk-size 64M \
--checkers=16 \
--dir-cache-time=160h \
--cache-info-age=60m
i mounted 3 drives with over 1tb of media
never run into any issue
i use it for jellyfin tho
You might have a point!
But maybe there's vital changes in how GDrive and pCloud and Koofr handle things ... (?)
Is this on Google Drive/Suite?
@chocolateshirt: Found the
--rcoption mentioned under Non Backend Flags:"Enable the remote control server."
... I do wonder what this means, though.
--rcI added it to mount to make a remote control on it.
How "Run partial scan when changes are detected" work.
I will give you an example:
1 -- First you added several album of
Avril Lavigneon Music folder, the tree will be like this:2 -- You scan it and plex index all the albums of
Avril Lavigne. Then you added another album of Avril and other artist, so the tree will be:3 -- On the next scan, Plex will scan only
Avril Lavigne - Album D,Avril Lavigne - Album E,Justin Bieber - Album A, andJustin Bieber - Album B. It will not scan unchanged files/folders, so it will not re-read/download the existing/scanned music. You can save a lot of bandwidth with this step.⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.
@chocolateshirt What I didn't understand is what a remote control means in this setting ...
Also, I tried adding dirs as you outline, but Plex didn't seem to pick up/be notified of the change and trigger any partial scan ...
with remote control, you could adjust some parameter without restart the mount services. About folder change, could you try to
lson the mount folder to make sure the rclone mount update the folder/files list. If there is no update then plex will not scan new file/folder, because it is not yet exist.⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.
mounted 3 disk, one gsuite 2 onedrive
rclone will auto recgonized if the flags isnt usable and will ignore if there is any N/A flag
should work with any drive, i tested onedrive gdrive and mega
i schedule my server auto scan media 3 times a day. mount and unmount disk 3 times a day as well
Just curious, why? Run into issues of some sort?
I've left my mount untouched and active for months at a time without any issues.
Humble janitor of LES
Proud papa of YABS
too frequent upload to my drives and cant wait for it to refresh itself
i wonder should i make the cache time lower too?
Not sure about other services, but GDrive over rclone should pick up changes right away unless you're doing some aggressive caching that I'm not aware of. I often upload stuff and run a scan immediately after without missing any of the new additions.
Humble janitor of LES
Proud papa of YABS
Oh, really, that's nice! How do you adjust parameters, then?
(So far I have stopped plexmediaserver, unmounted, fixed the mount script, remounted and restarted plexmediaserver.)
Changes are visible, when I add a folder, I can see it on the VPS where it's mounted using
lsetc.But it doesn't seem like Plex is notified. (Not sure which notify mechanism it would use/try to utilize, either ...)
Anyways, no big deal, I'll just hit the button to scan when I have added stuff.
And now, it seems everything works smoothly! When I rescan now, it's really quick, just picking up if there's any changes.
My current options are:
Seems to work like a charm now, and even when I selected all albums and hit Refresh metadata, it only used ~130 GB of traffic. (The library is now 142 GB.)
Not sure which of the options did the trick, but I'm happy.
Thanks guys! (Especially @AnthonySmith, this VPS has awesome performance!)
And BTW, PlexAmp now has loop/repeat option, and the regular Plex app also has the option to shuffle and unshuffle the running queue. (Only thing I'm missing in PlexAmp now.)
Plex's scanner is still quite buggy, especially with non-ascii 7 bit characters, it seems. (But for most a refresh of metadata fixes it.)
Yes, you need manual scan everytime you added new folder. I never used automatic one, hehe.. I am glad your mount working properly now.
⭕ A simple uptime dashboard using UptimeRobot API https://upy.duo.ovh
⭕ Currently using VPS from BuyVM, GreenCloudVPS, Gullo's, Hetzner, HostHatch, InceptionHosting, LetBox, MaxKVM, MrVM, VirMach.
Late to this thread. Currently have 180TB in GSuite (~75000 episodes, ~9000 movies, ~106000 songs), utilizing multiple Shared Drives instead of My Drive to exceed the daily limits via Service Accounts. My average movie is around 8GB, which utilizes the recommended bitrate of 8mbps - 12mbps for a 1080P stream. You don't need to set a lot of VFS flags anymore, as Rclone has them set by default. Here's my current mount command:
Regarding media scans, if you're using Lid/Rad/Sonarr, use Autoscan, and disable all forms of automated library scans within Plex. This will send scan jobs to Plex for individual folders when a download is imported.
Plexus: A suite of tools to help manage your media collection.
oh, i never know gdrive can do that! i used mainly onedrive and dont seem to pick up update and usually need a remount...
@Mason any idea which rclone flags help make that work?
Get the best deal on your next VPS or Shared/Reseller hosting from RacknerdTracker.com - The original aff garden.
Not really sure, but if I had to venture to guess, it may be the "--poll-interval" flag that @Wolveix has in his mount command above that'll do the trick.
Just make it low, like the 5s he uses
Humble janitor of LES
Proud papa of YABS
Thanks for your reply, I'll try that when I get the chance and report back for science.
Get the best deal on your next VPS or Shared/Reseller hosting from RacknerdTracker.com - The original aff garden.
For cleaning up metadata, do you just fix the tags in files and rescan the library, or do you delete/move away files, rescan library, fix tags and re-add and then rescan library? (With a much slower VPS, the latter was the only option that seemingly worked.)
Should be able to update the tags, then hit the 'refresh metadata' option for that song/album/artist/whatever fidelity you need (even the whole library, but that would take awhile)
Humble janitor of LES
Proud papa of YABS
I'll give it try
It failed previously, but that 512 MB vps struggled on some operations ...
(Did have plenty of slow HDD swap.)
Oh, you running all of this on a low end 512mb vps? Whats the streaming experience for you like? I've wanted to do the same since I don't need the transcoding, but I'm not sure yet about the performance and latency on the lower end.