Ark Clusters Across Multiple Machines

I have supported multiple servers for games for many years.  I got started because when I was in the Navy we had no internet connectivity so playing multi-player games meant someone had to learn how the servers worked.  Ark: Survival Evolved uses game servers and the configuration of these can become quite complex, especially since the game is feature incomplete.  I had been running multiple 'instances' of ark servers.  Different maps and different settings, I had carefully learned how to migrate people's profiles to the other servers using the underlying File System.  Then with Scorched Earth we learned that we could have these instances linked.  This changed things tremendously.  

What follows is a technical discussion about clusters, and its not a straightforward step by step how to set things up.  If you don't know what network file systems, symlinks and swap is then its probably going to be difficult.

Clustered servers means that you can have all 3 maps running simultaneously and move items and dinos and even players between them seamlessly.  Amazing!! this effectively changes the size of playable maps if you can just go to another map!  This means that the limit of the cluster is only a limit of hardware.  It takes a little bit of metal to accomplish this though:  The Island map with no mods runs at about 5.5 GB of RAM, The Center which is optimized and foregoes some visual candy can run at about 4GB even though its a much larger map.  Scorched Earth which is the newest map uses about 2.2 GB of RAM.  This means that to run everything we need about 12GB of RAM.   Individual client connections also put more load on the RAM and we absolutely want to prevent things from going to swap... but that's is a lot of server for a cluster, and it becomes more difficult once we start adding mods and their RAM requirements on top of things.  

To deal with the hardware requirements I simply switched servers on and off as required.  This was fine for the 3 or 4 people that were playing in my cluster as we pretty much wanted to be together on one server or another.  Till the holiday events. During the events we wanted to split up and collect as much of the costumes and swag as possible.  This presented a challenge. The Center is where we had our main base, since we took over one of the in game obelisks it was convenient for a base as there is room to expand around the base of the obelisk.  The Center did not have the holiday mechanics properly plumbed and so besides the holiday drops we were not seeing the DodoRex spawn, for that we would need The Island map.  Finally, there were the event's Scorched Earth features that only went off there... so we must have all three maps running.... i have plenty of machines that can do these maps individually; older hardware in the 4 GB or 8 GB range, but I only have one machine with 12 GB and that is my laptop.  How can we make this work?  


This is how!!!!!

The cluster directory on Linux is here:


I have seen on several tutorials where people that wanted to run multiple servers with different game settings have had multiple installs so they would look like this: 

/home/user/server1Location/Saved/clusters/clustername  /home/user/server2Location/Saved/clusters/clustername

with the clustername directory linked via symlink.  This would essentially allow to different servers to share items through the in game mechanics.  In case your wondering, items loaded into the obelisk in game are dropped into a file that is named what your steam profile ID is.  You could manually move this file to another server/cluster if you wanted to, but i wanted this to work without hand holding from the admins.  Since symlinks work I hoped that UNIX/Linux's Network File System would allow to separate computers to accomplish what a single computer was completing.  I tested it this morning and it does in-fact work!!!  The red arrow in the diagram above which runs directly between the two bottom computers is the NFS mount.  Now multiple computers can share this allowing for hundreds of servers simultaneously running in the same cluster.  I have not tested this on Windows since the NFS protocol is not baked in.  I might be able to do this with Samba/Windows shares, but windows is not set up for this distributed computing the way that Linux is, the NFS mounts do not appear differently than locally mounted devices to the Linux filesystems, and this is why it works so well with NFS. 

There are some drawbacks however.... your player profile was available to all the maps on a cluster that shared the same server directory.  (not everyone knew this but you could just login to new server and select 'respawn at random location' and viola your stats, levels and engrams were all available.)  Now since the server directories are different your player profile will be separate.  This is kind of a downer, but can be hand copied by the server admin provided they want this crazy task.  I suppose we could set up a sync between the servers and only copy the most up-to-date files.  I can think of couple programs that would be very good at this.  It also opens the door to cheating.  I dont think the developers ever intended to have the avatar profile existing in multiple instances across server but sharing a XP pool, but that's what has happened. 

The cheating would work something like this:  lets say I had an idea of the 50 most useful things that i always wanted to be omni-available in a obelisk.  I could load those things into the obelisk in game, then copy that file (the one with my profilenumber) into a third directory out of game.  Then I would simply write a shell script that copied this file back over the cluster file ever minute.  Then i could just hang out by the obelisk and download Ascendant Rocket Launchers and ammo every minute till I was overburdened with high powered death.... This probably would work for dinosaurs as well.  Mate Boosted 300 T Rexes?  sure you can get another 2 every minute out of the obelisk.  I have accidentally made copies of dinos moving servers around so I am quite sure that this would work.  

Next up I want to try and mount the entire game directory, then run multiple instances of islands with alternate save directories off the same game installation.  This would prevent some of the problems i already mentioned because they would all be working off the same save directory except for the files that are specified in an 'altSaveDir' setting upon start up.  It would also mean that only one server installation had to be updated instead of maintaining multiple working copies with synchronization. 


Why would i want to do this?  Digital Ocean offers a $5 a month rig that would server as the top level of the diagram and then the subsequent 'process' servers have an associated cost to run hourly... there is a 16 GB server that only costs $.18 an hour.  This means that running the server a 6 hours a day (which is what we are averaging currently) would cost $32 a month.  As long as I wrote a program to turn off and on these servers.  We might be able to use other cloud compute providers to make our ark cluster truly expansive.  Having a fiber backbone for the network would be infinitely better than my home connection and then we can really play with our dinosaurs in a way that is reflective of the digital age.  You know for fun.  

I can confirm that mounting the entire installation does work!!


All four maps on my cluster running simultaneously out of the same installation across 3 machines!

This means that as long as you have computers available you can spin up a new instance without actually configuring and installing everything on an individual machine!!!!!!!!!!!  More to follow!