Running a private image server
in Technical
I am working on a project which will deal with "many gigabytes" of images. Generally, the images will be static with an approximate turnover of about 30% of the images per year. There will be no user supplied images. The server will need to crop or stitch images together, but no processing beyond that.
Traffic is very low right now - since it's just me developing the project.
Is this something a low end box can handle for the time being? I don't want to be a noisy neighbor.
Comments
I don't think that it will be any problem. However, you can get a VPS with dedicated cores so you can run image processing for a long time without being suspended by the provider. There are many dedicated vps providers such as php-friends, hetzner VDS, server-factory for $5-10 per month
For example, a deal on LES at the moment: https://lowendspirit.com/discussion/3957/50-off-ryzen-epyc-vds-nl-alipay-btc-cc-pp-unmetered-incoming-traffic#latest
My concern with that offer would simply be storage. Realistically, I think I'd need something in the tb range, which is more in line with the storage servers which get advertised. I'm not sure my needs would be compatible with most storage plans. I did see VirMach possibly had a compatible plan, but they're out of stock.
and..
OP next time be more specific in your requirements versus wish list.
First you mention traffic, then "noisy neighbour' i.e high cpu resource use hinted.
Second post you mention storage space.
Next you will mention "I need more RAM"
all for LE Price. Of course, you do not think it is material to mention the budget, so good luck to you. And even better luck using a low horsepower system that you mentioned.
On a Hostcram VPS @Shakib I manipulated > 75 K Images, avg. 10 Mb in size,
transformed jpg to png (and reverse)
and all imageas to avif and webp.
Single disk of ~ 75 GB, trick was to batch the process and send it offsite storage server which can be set up as a cron job
If you are looking for an all in one solution, then LE might not be best approach,
Unless.. a BuyVM with a slice, or a Wishosting 4 Core Ryzen that one can find in their daily deals occasionally. Has enough RAM, BW and 500 odd GB SSD to take care of what you plan to do.
VPS reviews | | MicroLXC | English is my nth language.
Should be fine if you space it out a bit & don't process the entire TB lot in one go
How about localhost?
Action and Reaction in history
I think that you need spend more time on system design. For a large number of data, it will take too much time for a single machine to process. Therefore, it makes more sense to use multiple machines and process them in parallel.
If I was you, I will store all images in a object storage (Backblaze S3) with Cloudflare CDN to save bandwidth. Then, make a task queue to distribute the image url to multiple VPS. You can send a batch of multiple images to a VPS, a thread on VPS takes care of downloading the images to SSD and another thread processes them in sequence.
The "hostname" of a local computer, or the LES user who has that handle? :-)
VPS reviews | | MicroLXC | English is my nth language.
Well I was hoping to get some suggestions on what the requirements might be from people who have done something similar in the past. I know people use LES boxes for hosting static files and small PHP scripts. I don't know cropping and serving images through PHP/nginx is appropriate for a LES box, because I don't know how it compares to the average low end box user.
If this project is better suited for a kimsufi dedicated server, that's what I'll do ¯\_(ツ)_/¯
IMO as @havoc said you should be fine, as long as you do not overuse the CPU on a storage plan. Since you mentioned you'd need TB-scale storage, one sample low end option would be this.
I'm unsure how you'd be doing the cropping and whether you can actually limit the CPU usage effectively, so in case you can't, but would nonetheless like to go full throttle with the CPU on the cheap, additionally get this as your server (it's no Porsche, but it's amazing in terms of dedicated CPU computing power/EUR), then mount the one above using e.g. sshfs and do your image processing/serving here.
If you want it done fast enough, then a KS might be indeed your best option. (Alternatively, take a look at OneProvider Paris dedis)
Contribute your idling VPS/dedi (link), Android (link) or iOS (link) devices to medical research
Maybe hetzner cloud, with block storage, or @Francisco BUYVM with block storage?
Signature currently under construction
Sorry for the inconvenience
Right now my plan is to use the gd library and PHP to display a small part of a very large picture. Essentially using copying the example found in the php manual for imagecrop, but sending the output to the browser instead of creating a new file. Cropping would only happen on demand instead of trying to crop all images at once or in a large batch. The original file wouldn't be modified.
There is a handful of processing that I need to do to the pictures, but that would happen on my local system before they're uploaded.
I hadn't heard of oneprovider before. Thank you for the suggestion.
I already have some BuyShared products. It might be worthwhile to stay within the same company as much as possible. Maybe Francisco can chime to say whether or not the slabs are elastic - or if it's easy to go from 256gb to 1tb on demand without having to copy everything from one slab to another.
Cloudflare has everything you need:
It's more efficient than BuyVM because the processing occurs at the edge near the viewers.
I have a whole bunch of images on a local disk which need to be processed before they can be sent to a storage VPS. Is there a "reverse sshfs" which would let a remote cloud server (at vultr or similar, not a $20/yr VPS) access a local data on demand instead of waiting for large data transfers to temporary volumes?
Syncthing
Hetzner CX21(€ 5.83) + Hetzner BX11 ( € 3.45 -upgrade when needed) = great
Heck, you could, even if it is only while you are developing it, use Oracle's free 4-core ARM + 24GB RAM free offer to really crunch things & then RSYNC -> to wherever.
I'm using the Oracle cloud for other parts of the project and a few other things. Problem with Oracle is the free tier storage is no where near enough to fulfill the requirements. I really like how I can use Oracle's stuff without giving them a CC, so, I don't want to change that by starting to pay them..
Looks like I also missed out on BuyVM's stock for the month. Doh!