My Family Minecraft Survival Multiplayer Server built on AWS
Last month (November 18, 2021) marked the one-year birthday of my family Minecraft Survival Multiplayer server.
After over 1,500 hours of combined play time and 11 players at peak, I thought I’d share details on the setup and what I’ve learned so far hosting an active Minecraft server on AWS.
Some technologies I use include AWS EC2, ECS, S3, EFS, CDK, CloudWatch, Docker, Typescript, and PaperMC. One day I hope to release these AWS CDK Constructs to Github, so look out for that!
Background
In the winter of 2020, like many other people in our lives, my wife and I were in a rut of depressing news, pandemic anxiety, work stress, and local unrest in Seattle. Cut off from typical face-to-face social interactions at work and being unable to hang out with friends on the weekends, life was lonely and weird.
Enter: Familycraft 2020!
Looking for an escape and an excuse to hang out with loved ones for a dose of normalcy, I asked close friends and family members if they wanted to play Minecraft together over Thanksgiving holiday break.
As a software engineer familiar with hosting Java applications professionally, I also took it as an opportunity for a fun side project to host the Java-based Minecraft Server software myself, with a “backup plan” of renting a server from Apex Minecraft Hosting or similar if it didn’t work out or I lost enthusiasm.
Self-Hosted Minecraft: Requirements
Having hosted two Minecraft servers prior, a private world with my wife and a Happy Hour server with co-workers, both on a single cheap AWS EC2 instance, I had learned a set of requirements if I was going to host a world that others would cherish.
For “Version 2” of my attempts of self-hosting Minecraft, I wanted the following:
- Automatic, frequent, durable backups of the world in case of disaster
- Lots of RAM, a fast CPU, with 4 or more cores for minimal lag when up to 8 players connected
- Fast networking (players were scattered around the US)
- A Minecraft server version optimized for many players that receives frequent updates
- Hosting infrastructure that was “one-click” simple to tweak
- A locally-reproducible method of running the server so I could test new Minecraft versions, plugins and configuration changes on my PC before pushing them to “production”
- An easy way to instantly “pull the plug” to save on hardware costs if people stopped playing that wouldn’t interfere with my other side projects
- A voice channel and chatroom to rouse anyone interested in ‘crafting (Discord)
AWS: Go With What You Know
Being familiar with AWS professionally, I opted to go with what I know best. I knew EC2 would offer instances that could achieve the low network latency I needed for cheap enough, and S3 was an obvious world backup solution.
Docker: Packaged, Reproducible Runtime
To run, the Minecraft server software needs a Java installation, a .jar file, and a writable directory for the world files, which seems easy enough.
To achieve my goal of being able to quickly run a carbon copy of the Minecraft world on my PC to test out new plugins, Minecraft versions, new EC2 instance type, or do any world file surgery, I chose Docker so the server was runnable as one neatly wrapped package.
After getting half-way through building my own Docker image, I found an outstanding open source version that did everything I wanted and more. The itzg/docker-minecraft-server image acts as a configurable wrapper around the Minecraft server software, using environment variables to configure plugins, mods, admins, seeds, and version. Kudos to itzg for the awesome work.
I also found a configurable cronjob-PUT-to-S3 Docker image that would suit my needs for regular backups.
AWS CDK: One-line Infrastructure Updates
A challenge I had with a previously hosted Minecraft server was it was hard to change the instance, instance type, and JVM parameters powering the world.
This meant as the world got bigger and wanted an instance type with more RAM to run without lag, I needed to manually zip and scp files around, manually reconfigure firewall settings, reconfigure backup cronjobs, etc.
To avoid that headache this time around, I decided to spend the time up-front and to use AWS Cloud Development Kit (CDK) to define all of my Minecraft server resources as code and check them in to Git. This makes it simple to tweak things like instance type, make a “carbon copy” of the server, test out new instance types or versions, or roll back if something broke.
After getting a messy version to work, I broke up the stack into subcomponents, called CDK Constructs, to allow me to run multiple Minecraft worlds on a unique cluster or shared cluster, unique filesystem or shared, depending on what I wanted to do, and also to allow me to reason about components individually.
CDK Constructs: Logical Pieces of Minecraft Infrastructure
Logically, I was thinking about the infrastructure in the following pieces:
- WorldCluster: a collection of EC2 compute resources to run the Minecraft server instance
- WorldPersistence: a collection of runtime storage resources
- WorldService: the Minecraft game runtime
- WorldBackup: a component that periodically backs up the WorldService’s game files
- WorldConfig: a collection of per-instance configurations that are unique to the game — eg: the list of Admins, the Minecraft version (1.17, 1.18.1, etc), a list of plugins/mods, EC2 instance type, RAM
- World: Wraps all the above into one package, allowing me to easily add more server instances with a few lines of code.
- Stack: A collection of Worlds (maybe a Universe would be a better name?)
A World ends up looking something like this:
..and if I ever want to add another server for testing, another group of people, playing with a new Minecraft update, etc, it’s a one-ish-liner to the stack:
..and a one-liner to update the stack
Looking under the hood, each logical component has several AWS resources that interact with each other.
Making Changes: CDK Shines
A few months into Familycraft we decided to defeat the Ender Dragon. By that time, our server had gained a few new members, and we wanted everyone to be online together for the big boss fight.
To prepare for that event, I bumped up the EC2 instance type from a t3.medium with 1GB RAM dedicated to the game to a more powerful c5a.large with more CPU cores, a dedicated processor, 2GB RAM and faster networking, which was as simple as making this small diff then rerunning cdk deploy:
It turns out that this was a good idea! During the event, in the hour-ish that everyone was in the same few chunks during The End Dragon fight, the CPU spiked up to > 47%, which might have overwhelmed the t3.medium.
Minecraft Server version updates were almost as simple, bumping the “VERSION” parameter from 1.16.4 to 1.17.1, and utilizing the hard work itzg put into getting the right Java versions set up.
In Case Of Emergency: World Backups
To prevent against any in-game disasters, glitches, etc, or my mistakes when hosting the server, I needed a ‘fire-and-forget’ backup system.
To achieve this, the Docker image peterrus/s3-cron-backup runs as a sidecar to the Minecraft Server Docker image, peeking at the Minecraft World’s runtime EFS volume with read-only permissions, throwing them in a zip, and pushing to S3.
The image operates on a cronjob of every other day at 7pm which balances storage costs with recovering in-game projects. I’ll also occasionally manually trigger a backup after someone completes something epic. Again, CDK really shines here, allowing me to configure the backup image and schedule using auto-generated resource names.
Optimized For Multiplayer: PaperMC
In previous worlds I had used the vanilla Minecraft server jar directly from Mojang. These ran “okay”, but I saw frequent micro-stutters, rubber banding, and ghost blocks (where the client thinks there’s only air in a voxel but the server has a block), and after some Googling I found a few different versions of optimized server instances that claimed to have better performance when multiple people were online.
There’s a hilarious number of choices to pick from, including Spigot, PaperMC, Forge, Fabric, Airplane, Purpur, etc, etc, and I didn’t want to try them all. I read an article by Maddy Miller suggesting PaperMC was the best, and after making some tweaks (re-enabling TNT duping), it definitely runs more smoothly than vanilla.
This doesn’t come without a cost though. For example, Minecraft 1.18: Caves and Cliffs was officially released Nov 30, 2021, however as of writing (Dec 16), PaperMC has blocking bugs, which was a disappointment to have to wait.
Docker Run: Testing Dangerous Changes Locally
When Minecraft releases a new version, or there’s a new plugin/tool we want to try, it’s extremely useful to run the server locally on my PC before pushing those changes out to everyone that plays on the server.
To do this, I download a recent automatic update and run a few commands to start the server
and then configure the client to connect to localhost
..then I run around the world, which is both served by and running on my laptop, and play around with the important farms and areas of the world.
This was crucial recently, where I discovered some breaking bugs in the 1.18 PaperMC update that would have griefed everyone.
Craftin’ Tonight? Player Join Notifications
One kind-of awkward part of playing with a group was getting people to join around the same time to play together.
We use Discord for voice and video calls while we’re playing, so I spent a few hours writing an open source Spigot/Bukkit/Paper plugin, minecraft-player-notification, that sends a message to a Discord channel whenever a player joins a Minecraft server.
This nudges folks that someone’s around, and to pop in and say “hey” in-game.
I built this with all intentions of it only ever being used by our server, but to my delight it’s been used by something like 30 others as well!
The plugin works by intercepting PlayerJoinEvents, and using HTTP POST Webhooks to notify a Discord channel.
It has a few configuration knobs, changed with a config file like this, and it would be easily extended to other notification types, like rich Discord messages, SMSes, etc.
Fiddly Bits: ECS + EFS = :(
The most annoying component to get set up was the runtime world persistence. Getting EC2, Docker, ECS, and EFS to play nice took the most time of anything, and required a lot of trial and error.
The difficulty was in getting the Docker Container to allow connections to the EFS service backend, which required BRIDGE network mode and a few in/out ports punched in the EC2 Security Group.
I wasn’t able to find much documentation or prior art of how to achieve this in CDK, so this took a long time to figure out.
What’s next?
One thing I want to add to the setup is a CloudWatch alarm-based auto-sleep system to save a few bucks per month on hosting. Whenever there’s a period of X hours with nobody connected, the ECS service should be scaled down and the EC2 instance paused. The only reason I haven’t done that yet is to make sure the server is available for players when they want it, so I also want to build a simple web portal that players can use to turn it on without my help.
Another thing I’d like to add is DNS. Occasionally when making updates to infrastructure, the EC2 Autoscaling Group decides it needs a fresh EC2 instance, which rotates the IPv4 address. To support that, I’ll eventually get around to a Route53 DNS address and subdomain for the Minecraft server.
Appendix
Familycraft name
We call our family Minecraft server Familycraft, and were unaware of “Familycraft Dad” at the time. We’re not associated with him in any way, but he seems like a cool guy.
