Arrowhead's CEO says they have some "great support from top Sony backend engineers." Plus servers being raised to accommodate 800,000 players.

ethomaz

Rebolation!
21 Jun 2022
8,313
7,021
Brasil 🇧🇷
PSN ID
ethomaz
So they choose the optimization of their Server (that is already limited for upgrades) than go on horizontal upgrades (add more servers).

I can understand that because going horizontal needs to recode all the server net code to allow communication / syncing between servers and that is really hard to code and test.

Optimize their net code can double or more the numbers of sessions their server can hold simultaneously and doesn’t require a full recode of server net code… just redo the critical parts.
 
  • brain
Reactions: Gods&Monsters
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
Moderating
28 Jun 2022
18,418
15,400
So they choose the optimization of their Server (that is already limited for upgrades) than go on horizontal upgrades (add more servers).

I can understand that because going horizontal needs to recode all the server net code to allow communication / syncing between servers and that is really hard to code and test.

Optimize their net code can double or more the numbers of sessions their server can hold simultaneously and doesn’t require a full recode of server net code… just redo the critical parts.
They needed more servers period, now that Thyve added we are good. Optimizing servers they already had in place would have not solved the issues. It was a capacity thing
 

Yurinka

Veteran
VIP
21 Jun 2022
5,812
5,005
So they choose the optimization of their Server (that is already limited for upgrades) than go on horizontal upgrades (add more servers).

I can understand that because going horizontal needs to recode all the server net code to allow communication / syncing between servers and that is really hard to code and test.

Optimize their net code can double or more the numbers of sessions their server can hold simultaneously and doesn’t require a full recode of server net code… just redo the critical parts.
They are both optimizing code and adding servers.

Both things allow to include a bigger maximum CCU count, but optimizing is cheaper because allows to have more users per server. Meaning, they need less servers/space/server costs for the same amount of players.

Both things also help to minimize saturation related issues. Some saturation issues can be solved by both, while other ones can only be solved by one of them.
 

ethomaz

Rebolation!
21 Jun 2022
8,313
7,021
Brasil 🇧🇷
PSN ID
ethomaz
They are both optimizing code and adding servers.

Both things allow to include a bigger maximum CCU count, but optimizing is cheaper because allows to have more users per server. Meaning, they need less servers/space/server costs for the same amount of players.

Both things also help to minimize saturation related issues. Some saturation issues can be solved by both, while other ones can only be solved by one of them.
They didn’t add server.

They did vertical upgrade that is increase CPU core and RAM… that allowed them to increase the max sessions from 250k to 360k.

Now they could have started to work in the net code to allow multiple servers syncing… but you will only see that in the long term.

Optimizations already give results while finish the net code for multiple servers will take months yet.
 
Last edited:

Yurinka

Veteran
VIP
21 Jun 2022
5,812
5,005
They didn’t add server.

They did vertical upgrade that is increase CPU core and RAM… that allowed them to increase the max sessions from 250k to 360k.

Now they could have started to work in the net code to allow multiple servers syncing… but you will only see that in the long term.

Optimizations already give results while finish the net code for multiple servers will take months yet.
No, increasing the CPU and RAM doesn't increase the CCU limit. With that you'd get more sessions per second or minute, not more at the same time. At this point what suffers the most is the HDD/database, which I assume they already maxed out per server. Also, they are Sony, so obviously got the top tier available since the start.

So to grow the CCU they have two options, to add more servers or to optimize the code.

According to their CEO they added more servers to increase it to 800K and obviously keep working on optimizations and fixes, optimizations and fixes which in most cases we won't notice until the next update.

The servers are obviously synched because since like 2 decades ago servers are in a cloud and if not synched -even if they work independently each one working with a different group of players- the game wouldn't work.

Optimizations help to reduce the number of connections made per user and their frequency, or to reduce the database, CPU, RAM or HDD needed per user/session, or to reduce the ping, or to rebalance, or to distribute the workload better between different databases, or servers or many other things. Or to "move" certain things more when makes sense to Akamais or similar to reduce server stress.

P.S.: I worked in games with over 3M DAU, I don't remember the CCU limits, I'm a coder but a gameplay (client/frontend) one. Some of the guys from work with who I hanged out (and with some still) were the server programmers of my games. They now work as lead or expert programmers at Ubisoft, King, Scopely or CoD Warzone mobile, games much bigger than this one. That team was so good that the leader of the team now is the studio head.
 
Last edited:
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
Moderating
28 Jun 2022
18,418
15,400
They didn’t add server.

They did vertical upgrade that is increase CPU core and RAM… that allowed them to increase the max sessions from 250k to 360k.

Now they could have started to work in the net code to allow multiple servers syncing… but you will only see that in the long term.

Optimizations already give results while finish the net code for multiple servers will take months yet.
You are incorrect, they added servers and tweeted as much.
 

ethomaz

Rebolation!
21 Jun 2022
8,313
7,021
Brasil 🇧🇷
PSN ID
ethomaz
You are incorrect, they added servers and tweeted as much.
Can you point it to me please? Because the only post I saw about adding more servers is in a metaphor way and talks about how they have to "rewrite" their managers to sync the servers.

But perhaps they already rewrite the syncing so it is all good.
 
Last edited:

ethomaz

Rebolation!
21 Jun 2022
8,313
7,021
Brasil 🇧🇷
PSN ID
ethomaz
No, increasing the CPU and RAM doesn't increase the CCU limit. With that you'd get more sessions per second or minute, not more at the same time. At this point what suffers the most is the HDD/database, which I assume they already maxed out per server. Also, they are Sony, so obviously got the top tier available since the start.

So to grow the CCU they have two options, to add more servers or to optimize the code.

According to their CEO they added more servers to increase it to 800K and obviously keep working on optimizations and fixes, optimizations and fixes which in most cases we won't notice until the next update.

The servers are obviously synched because since like 2 decades ago servers are in a cloud and if not synched -even if they work independently each one working with a different group of players- the game wouldn't work.

Optimizations help to reduce the number of connections made per user and their frequency, or to reduce the database, CPU, RAM or HDD needed per user/session, or to reduce the ping, or to rebalance, or to distribute the workload better between different databases, or servers or many other things. Or to "move" certain things more when makes sense to Akamais or similar to reduce server stress.

P.S.: I worked in games with over 3M DAU, I don't remember the CCU limits, I'm a coder but a gameplay (client/frontend) one. Some of the guys from work with who I hanged out (and with some still) were the server programmers of my games. They now work as lead or expert programmers at Ubisoft, King, Scopely or CoD Warzone mobile, games much bigger than this one. That team was so good that the leader of the team now is the studio head.
Sorry I think I read it wrong.
Are you implying that adding more CPU and RAM to their database servers doesn't increase CCU limit?

According with the CEO they have optimized their server net code with the help of Sony to reach 800k CCU.

 
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
Moderating
28 Jun 2022
18,418
15,400
Can you point it to me please? Because the only post I saw about adding more servers is in a metaphor way and talks about how they have to "rewrite" their managers to sync the servers.

But perhaps they already rewrite the syncing so it is all good.
I actually think you might be correct. I could have sworn I saw a tweet about adding more servers but all I find are tweets about adding server capacity, optimization and scaling.

"It's not a matter of money or buying more servers," Pilestedt posted online in response to a player query. "It's a matter of labor. We need to optimize the backend code. We are hitting some real limits"
 

Yurinka

Veteran
VIP
21 Jun 2022
5,812
5,005
Sorry I think I read it wrong.
Are you implying that adding more CPU and RAM to their database servers doesn't increase CCU limit?

According with the CEO they have optimized their server net code with the help of Sony to reach 800k CCU.


I wasn't implying it, I was explaining it.

And no, nowhere in these tweets say that they increased the CCU to 800K because the optimizations.

In the first tweet he mentions they are reaching the 800K mark, doesn't mention anything about optimizations.

Later he replied to someone to deny that their servers programmers are "sacrificing their sanities to keep the game from completely collapsing under this demand" "just rewriting the backend", mentioning that instead are working mostly on optimizations, with the great support (those who probably are handling the most difficult issues) from Sony server coders, while the rest of the team (those who aren't server engineers) works in something else.

To increase the number of servers just requires a handful clicks (and get charged a lot of money), doesn't need more work beyond that. Until something else gets saturated and needs to be addressed, obviously. In fact the proces of increasing or decreasing the number of servers can even be done automatically, getting more or less when reaching certrain levels.

There has been several days without CCU saturation, which caused the issue we had the too long login waitlist issue.

So their work now is optimizing stuff to make sure certain specific things don't break, as they did with matchmaking or rewards, and I assume now are doing for those in-game stats they temporally disabled because got saturated and whatever is causing the crashes.

Optimizing you achieve things like reducing the HDD space required by the game in the servers, or to complete sessions/server requests faster, helpful to handel more sessions per minute/hour/etc. Which is different from allowing more concurrent sessions/users running in the servers at the same time, even if both things help.
 
  • brain
Reactions: Gods&Monsters

Entropi

Veteran
Icon Extra
22 Jan 2023
2,110
3,148
It was a matter of time and now cheaters are here, of course. Does anyone know if disabling cross-play fixes the issue?
 
  • Like
Reactions: Gamernyc78

Yurinka

Veteran
VIP
21 Jun 2022
5,812
5,005
Show us the cool stuff already!
It would be cool to see new stuff -as I remember they'll post stuff, maybe a new warbond, the second thursday of every month- but what I want is to see first is the frequent crashes on PS5 getting fixed.

I'm so fucking tired of them.
 
  • Like
Reactions: Gamernyc78
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
Moderating
28 Jun 2022
18,418
15,400
It would be cool to see new stuff -as I remember they'll post stuff, maybe a new warbond, the second thursday of every month- but what I want is to see first is the frequent crashes on PS5 getting fixed.

I'm so fucking tired of them.
I hvnt had a crash since last week I put in like five hours yesterday and all was well.