poyliquid.blogg.se

1. #EC2 LINUX IMAGEMAGICK HOW TO#
2. #EC2 LINUX IMAGEMAGICK INSTALL#
3. #EC2 LINUX IMAGEMAGICK UPDATE#
4. #EC2 LINUX IMAGEMAGICK CODE#

#EC2 LINUX IMAGEMAGICK HOW TO#

#EC2 LINUX IMAGEMAGICK INSTALL#

#EC2 LINUX IMAGEMAGICK CODE#

I'll tag all the folk who've pointed out the need for that in this thread when I do.Įdit: I'm going to instrument my code to report time spent waiting in thread synchronization functions when I do that.

#EC2 LINUX IMAGEMAGICK UPDATE#

I'm going to measure it all properly while controlling as many variables as I can and post an update thread in due course. I did not do anything this morning when I got those numbers that would qualify as good science. Maybe different ImageMagick builds are just that much faster and slower on each platform. Clearly thread context switching can torpedo performance on CPU-bound tasks.īut that could've just created a cognitive bias to explain everything in those terms. I actually saw bigger proportional differences from fiddling with that parameter on the same platform (Ubuntu) than I saw between different platforms. The reason I arrived at the kernel explanation is that I was in the midst of measuring the differences between restricting ImageMagick to a single thread per core, and letting it create threads as it saw fit but still spawning it a number of times equal to the machine cores. And I would probably have been in here commenting the same if someone else had posted this thread. Pure CPU speed differences for parallel processing tests on the same hardware (tested using threads running ) were observable with Ubuntu vs Amazon Linux when I tested, but Amazon Linux was only ~4% faster. Much later edit: I never got around to properly testing all of the isolated components that could've affected this, but as per discussion in the thread, it seems clear that the actual source of the huge difference was different ImageMagick builds with different options in the distro packages. But for this project, clearly, that extra setup work is worth it.

Ubuntu's generally a lot easier to get workloads up and running on. Of course, this all comes with a bunch of pains in the ass due to Amazon Linux not having the same package availability, having to build things from source by hand, etc. I'm not a Linux kernel guy and I have no idea exactly what AWS has done here (it must have something to do with thread context switching) but, holy crap. Speed on Amazon Linux 2: ~180,000 images per hour Speed on Ubuntu 20.04: ~30,000 images per hour

Speed on Windows Server 2019: ~70,000 images per hour

Task: Run ImageMagick in parallel (restrict each instance of ImageMagick to one thread and run many of them at once) to do a set of transformations (resizing, watermarking, compression quality adjustment, etc) for online publishing on large (20k - 60k per task) quantities of jpeg files.Īfter porting the Windows orchestration program that does this to run on Linux, I did some speed testing on c5ad.16xlarge EC2 instances with 64 processing threads and a representative input set (with I/O to a local NVME SSD). Just thought I'd share an interesting result from something I'm working on right now.