Sooooo,

I had to move one of my customer’s websites to a new server, because the old one was… well… pretty old, and didn’t support PHP 8. As it’s usually the situation with all the big hosting services, they don’t give a crap about you, so you’ll have to make the migration yourself.

And the configuration yourself.

And the debugging yourself.

As I initially thought, the migration of the servers went out pretty smoothly, especially when dealing with a Wordpress site.

The first thing to do is to transfer the files. Any noob would get a local copy of the server on their computer and then upload the files on a new server.

So instead of working hard, I had to work smart, therefore I had to make make a simple script. But before that I noticed there was a big issue. The new server couldn’t connect over SSH to the old server, getting a cute error instead

1
2

$ ssh root@10.0.0.69
Unable to negotiate with 10.0.0.69 port 22: no matching host key type found. Their offer: ssh-dss

The 10.0.0.69 IP is fictitious, don’t try to connect to it, you might make your microwave think it’s a real oven

So after a bit of research I found the following command, which seemed to do the trick.

ssh -oHostKeyAlgorithms=+ssh-dss root@10.0.0.69

But this didn’t work properly with rsync, so I had to take it to the next level and add a host pattern in my ~/.ssh/config file. Give your host a simple name, like oldserver, potato, or junkheap, save it and you should be good to go.

1
2
3

Host oldserver
  HostName 192.0.0.69
  HostKeyAlgorithms=+ssh-dss

Now that the rsync can properly connect to the server, time to create a .sh file, make sure we run chmod +x sync_sites.sh so we can run it (duh!), and then add to it the content in the snippet below. You can add more servers to the script, and if you don’t want to have a sense of progress of what’s happening, you can remove the v parameter from the -avzr, to make it less verbose, not showing you what files are being transferred. Always it’s a good practice to stop the script with Ctrl+C after running it for a few seconds, to ensure the synced files land in the expected directory. If they don’t, check your trailing / characters, that’s usually the culprit. Also a good practice is to exclude the wp-content/cache folder as we don’t transfer useless data.

1
2
3
4
5
6
7

#!/bin/sh
echo "Script started at $( date +%m-%d-%y@%T )  "
echo "Syncing Site Name"
rsync -avzr -r --exclude 'wp-content/cache/' -e ssh username@oldserver:/var/www/sites/mydomain.com/html/ /var/www/vhosts/mydomain.com/httpdocs
echo "Finished sync for Site Name"
echo "============================"
echo "Script ended at $( date +%m-%d-%y@%T )  "

Ok, now that the script is running, we can go make ourself a cup of coffee and read the news, while we monitor the file transfer progress. Or go for a walk with the dog, while using TeamViewer from your phone to check your computer screen. Or set up an ingenious system of video cameras, I’m not your boss.

At this point, once all files are copied, it helps a bit if you’re putting the site in maintenance mode and show a “brb” message. Because you’ll have to transfer the databases, to edit the config files and then you’re done.

Right?

Right!

Wrong.

Even if you switch from the same server type to the same server type (but newer), your OS choices might be limited and completely different from what you were initially using. And if you’re migrating from two different servers from two different vendors, then you’re SOUL (shit out of luck).

So basically there’s nothing that you can really do but open the website, turn the debug on and start smashing your head against the metaphorical wall of “Parse error:” and hidden errors.

And after editing and editing, you get a page that loads half way. And then you hit some more errors in the head. Sometimes it’s a big thing, sometimes it’s because in php8 the conversion of non-numeric string to number is considered a fatal error.

1

Fatal error: Uncaught Error: Unsupported operand types: string + int in /var/www/somefile.php on line 420

And then you edit some more file and eventually you get a fully working full page. And then you open a second page, like a post, and you find yourself back to square one. Okey, back to bashing the head against the “Parse error” wall.

Then log in to the Wordpress admin and, guess what? Yep, the wall again. Break that again and again and start testing stuff.

Uploads fail? Definitely permissions on the wp-content/uploads/ directory.

Updates fail? Definitely permissions on the wp-content/ directory.

Page gives a 403 Error? Definitely a rule in mod_security which blocks a page with more than 4 errors. Heh.

Once everything is checked and everything seems to be working fine, you take out a very big hammer and you give it to the site editors.

And when you iron all the errors that they find, which are none, because you did a good job at testing stuff yourself before saying things are done, after five days there comes that middle-of-the night email saying in a more polite way and with more words:

So let’s see what happens. I open the homepages and some random articles and they seem to load fine, so the cache layer is doing its job properly. Logging into the admin was another question.

First thing to check is always server space. Even with 400GB of disk space, you can never have enough space.

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       394G  394G 0B    100% /

Welp, this seems to be it. But the total of our sites doesn’t even reach 100GB in total, what on Earth could have happened that filled another 150GB of storage? Let’s see where we have a big directory, and usually the start points are either the web folder or the logs. The following command will scan the contents of /dir_name or whatever you give it and show you the top biggest folders and files.

1

du -a /dir_name | sort -n -r | head -n 10

I was pretty lucky, because I quickly found out that the bulk of the space was eaten by PHP error logs. I thought I’d see an error or two, but it seems the server was generating about 40 GB of server logs A DAY. Tailed into the most recent one and OMG!, it seems there’s this error that was being triggered about 12000 times PER SECOND, EVERY SECOND, ALL DAY LONG!

1

WARNING: [pool mydomain.com] child 42069 said into stderr: "PHP message: Connection refused"

This caused the log files to fill in very very fast, impacting both available server space and server performance (as it kept writing the log file). But if that’s the culprit, then what seems to be causing it?

A bit of searching on the interwebs revealed that in the server move, some settings are really NOT making the new server happy. In our case, it was telling the W3 Total Cache plugin to use Redis as Database and Object Cache. Once I switched those to disk, the errors stopped pouring instantly.

Additionally, one could choose to turn off the logs, just to avoid this issue in the future, but it’s good to also have them around. You can turn them back on anyway.

Oh, and if you were wondering, I deleted all 160GB of php error logs which compressed were only about 23MB. 🙃

This post is a part of the November Agora Road Travelogue, an effort to promote blogging.