COMMUNITY - FORUMS - GENERAL DISCUSSION
Server Stability Testing?

Good morning All!

So, we have seen some good information, and all the recent DJ's and blogs are pointing to testing beginning relatively "soon". The client is taking shape, features are being implemented, etc., etc. That being the case, there was one particular area that I have been concerned about for a while, and was hoping we have reached a point where it could be addressed more specifically.

Server stability.

To put my question in context, I am going to do something that I prefer not to do - compare CoE to another game. That game is Camelot Unchained.

CSE understood that in order to have large-scale combat, the server/client needs to be able to support hundreds, if not thousands, of characters on-screen simultaneously. Due to that, they decided to go completely in-house for their engine in order to ensure that it will meet their particular needs in regards to supporting the player density and performance they are looking for.

Now, I understand CoE is not CU - CU is almost entirely about large-scale PvP. That being said, there are a lot of factors in CoE, IN ADDITION TO large-scale PvP, that will encourage large amounts of players to congregate in limited areas (Sedecim is the first example that pops in my head).

So, all that being said, here is my question:

Can you guys give some insight into what you have been/are doing to ensure that performance is going to be where it needs to be in regards to high player density?

Despite all the great info we have received lately, and the obvious progress being made, I don't recall much being said on this subject.

As another comparison, this is practically the #1 thing CU discusses and updates on. It's what they addressed before they really even started adding features. Even before they had a "world", CU was testing performance via running hundred/thousands of "bot clients" simultaneously on their client, and tracking FPS, to see where they were at. I believe the last number I saw was in the area of around a thousand bots/players involved in simultaneous combat while maintaining reasonable FPS (I don't recall the exact numbers, but anyone is free to google the details).

On the flip side, I don't recall seeing one, real, mention or discussion about the issue from SBS.

To dig into the question a bit more:

  • Have you done any stress testing in-house to measure performance during "high player count" interactions?

  • If so, can you provide any numbers?

  • If not, can you provide some insight into when you intend to start doing this?

  • Are there any details you can provide that would alleviate concern in this area, given how prevalent it is in almost virtually every MMO.

I am not trying to say you guys haven't thought of this (I remember several comments about how the server will be set up to handle player load - Vye related it to using locality and "rooms"), I am just saying I don't recall any details or specifics relating to the subject in recent times.

Just hoping someone could shed a little light on what is arguably one of the most important aspects of any MMO's success (ESPECIALLY one on the scale/scope of CoE).

Thank you much!


Imgur

10/17/2018 2:05:20 PM #1

I am going to go out on a limb and assume "no comment"?

;)

Snipe did have this to say on Discord:

SnipehunterLast Monday at 1:20 PM

"Area of concern" culling is a pretty common practice these days: We generally arrange the game client in a way that it will only render what actually matters to it, meaning people in different rooms or plausibly out of view are generally culled for performance reasons. You can layer other optimizations into that same path so that a client's area of concern is limited no only by space and visibility but by activity, or game mechanic, as well. The net result is that literally you'll ever render what your character can see. Even a crowd can be optimized so that you only see the people in detail that are close enough to affect you, that sort of thing.

Tarvald HammerBullLast Monday at 1:21 PM

Good stuff Snipes, if we do see a community or a battle that has grown exceedingly large, will we see incorporated load screens or anything of that nature, even with the "Area of concern" culling.

SnipehunterLast Monday at 1:23 PM

Hopefully not, but we haven't actually implemented any optimizations yet, so its a bit premature for me to say. That does seem highly unlikely given the way we're architecting the game though.

I don't know that it really addresses my concern, as it is still referencing things that are "out of sight". Additionally, if "area of concern culling" were truly that common, and it worked, my logic tells me that fewer MMO's would suffer performance issues with lots of people on the screen.

I could be misunderstanding that.

Bottom line, I get the impression it is not something they are focused on specifically testing at the moment. Maybe, that is because they are supremely confident that, due to whatever they are doing on the tech end, they are sure it won't be an issue.

I guess we will just have to wait and see if that is the case. I truly hope it is.


Imgur

10/17/2018 7:24:52 PM #2

"SnipehunterLast Monday at 1:23 PM Hopefully not, but we haven't actually implemented any optimizations yet, so its a bit premature for me to say. That does seem highly unlikely given the way we're architecting the game though"

What I take from this, not being a computer person myself, is that the issue you are raising has been broached within the studio and they are aware of it's possibilities. But the current development stage has yet to address it.

It's a valid point, and one that I don't think they are taking lightly, if that helps at all.

But not knowing what exactly they are doing or not doing "under the hood" to address it is kind of hard to really know.

I'm looking at it like this:

When I take my car in for service, I trust that my mechanic technically knows how to fix the particular thing that I am concerned about without him having to go into the details about which particular wrench he's using to do it.

But that is also because I am so ignorant about it he could say he's using a "left-handed monkey wrench" and I wouldn't know the difference.

There are a lot of people that would though, lol. That's what it sounds like here. You are curious on how it is being done.

For me as long as the car runs like it is supposed to, I'm good.

How he made it run?...Well that could be voodoo for all I know.

So...I would kinda agree with your overall observation that it is not a specific focus...at this moment.

Should it be?...In the overall big game picture...Yeah.

Right now?...No freaking clue.

Not sure that helps anything at all, but at least you know somebody somewhere did read your post and is also concerned about it too.


We Are The Many... We Are The One... We Are THE WAERD !!!

10/17/2018 7:47:59 PM #3

Gather all your friends and bunch up in a small space as soon as Prelyria is available. That will at least give you - and SBS(!) - a little more insight into how optimized it is for this scenario atm.


10/17/2018 7:54:33 PM #4

I think a good way to answer this is with what Caspian posted in the 0.4.0 release post. Adventure Completion From Pre to Alpha

Our approach to development is something I've referred to in the past as our Rope Bridge Philosophy. Which, while I won't go into great depth here, is about building thin vertical slices of our systems and solutions - in isolation - and then iterating on them quickly and frequently. Our implementations focus heavily on developing with SOLID design principles in addition to a couple of our own principles. In particular, we try to develop CoE in a way that allows for "Separation of Concern", "Iterate Where it's Cheapest," "Code by Necessity," and "Optimize Last."

The idea is that while they are aware that the servers need tested and the game may need optimization efforts put in to make it so 300 people all in a small area is a viable play option, They aren't designing the game around fixing things that aren't broken yet.

They are probably aware of server requirements and are going to make sure they can handle the stress, but step 1 has to be make a good game with step 2 being make the game run smooth. "Optimize Last"


10/17/2018 8:43:47 PM #5

All I remember is they have a bunch of tech in the background required to make all this stuff work. Such as:

Docker Swarm

RabbitMQ

PostgreSQL

I aint no computer whizz so none of that means anything to me lol.


10/17/2018 8:54:25 PM #6

This is an interesting conversation, but I doubt CoE needs to achieve the same numbers CU aims at.

This issue is mostly important for PvPers. CU is a game for them. CoE also provides something interesting to PvPers, but is not entirely focused on that. The percentage of PvPers will be lower in CoE.

Also the land will be very large. It will be very difficult to gather a hundred of characters to one area. Maybe during an all front attack on a big city, but this might never happen. Skirmishes will happen in more regional areas (opinion here, not fact!)

If we take rough numbers we have, 100 000 players will be divided in kingdoms, duchies and counties leaving less than 100 players in a county. Many of them will be crafters and probably won't go to war.

This being said, they may need to stress test battles with a few hundred of players, but I doubt the engine will require what the CU engine will offer (if they get there!)


10/18/2018 12:24:41 PM #7

It is a long post dear fellows, not surprising to those who knows me. Starting with some other games who got/have similar problems, then checking on CoE's specificities.

Single shard worlds and activity concentrations

If we have a quick look at EvE Online, the safe (no-combat zone) tribal capitals are a server nightmare, because of two things: there is a lot, and i mean a lot, of people in close vicinity on the same local map and which are very active transaction-wise (transactions that each impact the whole world and its economy).

It regularly lead to waiting queues to simply enter the place.

Second is when there are those famous battles (guiness book records by the way) with thousands concurrent online players, but it is easier to handle with a predictive allocation of server resources... and they don't impact the whole world, just the local space-ography.

Optimize later?

Then you have similar titles who tackled the problem too late, like Star Citizen: despite their budget, they still have a hard time about just that, with a goal of 200 players per solar system, though I didn't follow their recent conference.

And almost every MMO out there, except maybe Dual Universe: since its core feature is a technology to handle a single shard server with any number of clients/players in close vicinity.

Culling limits, many

Culling was used as a last line of defense by Guild Wars 2 for both the RvR and event maps, trying to pass the 200~250 participant limit.

And they also had a really hard time trying to optimize that for a couple years after launch... to the dismay of their PvP crowd (among other things not thought when they should have).

Soulbond partnerships

So yeah, the OP question is a very valid one, one that was resolved early by SbS with their partnership with Improbable (massive network engine)... except it didn't work for reasons we can only speculate, so back to square one.

Unreal Engine is a very potent and experienced graphical engine (actually more like a full game engine), but doesn't do much for network, and definitely not for online requirements above the dozen.

On screen no-limit

In a sense, don't worry on how many "characters" a game may have simultaneously on screen, it is a concern of the past: graphic processors and memory can handle that just well, just like any other "destructible" or "interactable" asset.

Total War series can vouch for me as an example and repeated proof of concept.

Data exchange is the issue

The real problem is about how much information has to be exchanged among clients and between clients and servers: since in current and classical implementations, it grows exponentially as clients (both players and scripts) find themselves close to each others and their data/interaction bubbles cross paths.

And you can't rely on peer-to-peer, though it's "cheap" and done well can alleviate the issue, it leaves the door open for hackers and cheaters (as any PvP player would tell you, it has ruined many games that didn't have a dedicated server infrastructure).

Redeeming factors in favor of Elyria

An announced maximum of around 100k players per server (a bit less for OCE) in total.

In Elyria, local numbers are surprisingly not that big, density-wise... around 60 players per county in average from what I read on the forums (within an average 16km² if I recall correctly), and ten times that in NPCs (at most).

An elyrian city at max-level only needs a minimum of 250 elyrians in population: roughly an average of 3 people per parcel, while a single perfect parcel could hold 64 cottages side-by-side (on just the ground level).

Note that there is always the possibilty to join a server through a NTC code, but as player characters never disconnect, the total elyrian (PC, OPC, NTC, NPC) population to handle by a "server" stays relatively constant: that's smart.

A measure of "lots of people" in Elyria

I don't know, players will try to push the enveloppe.

But I can only speculate that it won't be that much... or to be exact, it will be highly difficult to gather a thousand characters in the same place (for a memorable concert or battle), and live for another day: resource depletion, body collision, elyrian time to do the trip... heck, they will probably not hear or see what happens on stage!

Business model to the rescue

Actually like any major website or online app, a "game server" is a myriad of hosted servers working seamlessly together... and generally linked with fiber optics (with next to no ping at all). If one fails for any reason (or is suddenly overworked), another kicks in and pick up (or share) the workload.

The Life Spark we use to spring life into one character for a RL year is our paying subscription to the online service and support. Rhetoric question, who pays for the NPCs? ;)

That's why the business model includes a ratio of NPCs per Soulborn (player character, online or not)... and I imagine that ratio will evolve during the development phase to match server performances.

In a nutshell

If they manage to solve the conendrum of data exchange between clients, servers, the game will be smooth.

If they don't, there are still the (solo/team) prologue and Discord-mode to enjoy. :p

Organic design

That's why an organic design approach is on the safe side: you get what you can "sense".

"Sense", not "see", that's a mistake that shouldn't be made: you can look at the horizon (40 kms away on earth I think) and perceive a sail or a landmark... that's things the engine will draw for you if you look that way.

Doesn't mean the character (and player) need to get updates from everything happening within that range, actually it shouldn't... graphical engines learned to do it the hard way, internet too, gaming network not so much.

What matters is on whatever someone focus their attention and their peripheral "vision" (ie. perception, including sounds and maybe smells if they are implemented), there are the only things that need constant data updates outside their own status.

The rest can be leisurely updated (at slower paces, like weather changes, etc.) For instance, people passing down the streets, you only need their "silhouettes", not their entire character sheet.

In that example, if you give a particular group a "second look", then the network/game-master engine will respond and bring you more details... and provide them with a possible hint that they are observed (skill checks, etc).

It is the same over-detailled data in the background, but a character (played or not) get only simplified views, just what was needed/determined in such situation/case.

It also make things easier to script behaviors reliably... something few online titles took time to look at, generally engrossed in theme parks and level designs instead.

And that is, my two cents on the topic, you can keep the change. :)


[EU/EN/FR] An enlightened mind, a wandering soul (or vice versa) : mayor of the fair city of Aurora Beach, civil engineer by design, inscriber wanabee, outfit maker enthousiast... and editor at the Skyfall:Archives.

"May the land be fair to you." (Friend Code : 48FDFB)

10/18/2018 4:30:22 PM #8

@Leonim

Excellent breakdown, thank you! You definitely have a level of technical expertise I don't possess.

When it comes to this topic, as concerned as I am about it, I often feel it comes across to technical people as the equivalent of, "Me think lag bad, make game have no lag". ;)

Thanks!


Imgur