Helldivers II Was Built on an Archaic Engine That You Can't Access Anymore

Darth Vader

I find your lack of faith disturbing
Founder
20 Jun 2022
7,365
10,933
This is non-sense, where I work we have a web app distributed on Azure, I assure you that Azure is no guarantee of performance.

The "talking to a database" part, like the network code part is normally bespoke as soon as there is a high enough level of complexity.
I never said Azure was a guarantee of performance, but people were blaming Azure for not being able to support the amount of users they had concurrently, which we both know is bullshit. They have much bigger clients and can cope with much bigger volumes, so that is just fanboy speak.

As for bespoke code, exactly, but your engine needs to be able to integrate that bespoke code. So here either the engine can't, that bespoke code is not scalable, or a combination of both. It can absolutely be an engine issue.

This is a very cool video that talks about a lot of high level concepts in development.


and this is very cool around scaling multiplayer games. There's a lot of PR in it of course, but it kinda goes through a lot.


I'm sure you know about most of this stuff already, don't take me wrong, I'm just dropping here out of interest
 
  • Like
Reactions: nongkris
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
28 Jun 2022
20,386
16,652
They pretty much confirmed that the game wasn't built for all those players. Botton line they didn't expect the success
No one would have for saw that that goes without saying. Either way your engine should be capable of worst/best case scenario. Especially when they know it's a GaaS game. WTF thy thought only 5,000 ppl would play. That's short sighted thinking and I doubt they thought something like that. Shit just happens.
 
Last edited:

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
Is it that complicated to fix? The Azure stuff must be the last thing they did. Why is it taking so long 😫
Net code is not related to the engine used.
And it is the hardest thing to test because it is impossible to have the same amount of user in a development lab (or even public open Beta) that you have after release a game.

Way bigger companies like Capcom, Bungie, Blizzard, etc all have to deal with net code issues after release.

I wish it could have a way to simulate the same scenario os millions of players in random locations before release but that doesn’t exists.
 

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
They pretty much confirmed that the game wasn't built for all those players. Botton line they didn't expect the success
I think the issue is more like…

They can’t test with the amount of users you have after the release.

Every bigger multiplayer developer have issues with net code after launch.
 

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
I wrote about this in the connections thread but the multiplayer scaling problem is most likely tied to this engine being so old. I can't recall a game in recent memory that a hard cap on number of players online.
Don’t you remember?

Halo Infinite had insane queues to enter in the multiplayer.
That was two years ago.

Let’s not go too back… last CoD was dozens of network issues around the launch even after a huge open beta… you can’t even connect to the lobby at launch.
 
  • Informative
Reactions: Gods&Monsters

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
You guys don't know enough about backend if you think game engine is responsible for server side problems smh...
Yeap…

I have no ideia why people are blaming the engine due net code issues. In any graphic engine you use you will having to create your own net code for online play.

If the engine works well locally or offline then the issue is the net code part and not the engine.
 

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
Sony most bugged game and troubled launch in the last decade was Days Gone, one of the few AAA of their developed using Unreal Engine.

Maintaining your own engine is likely very hard but maybe having the personnel capable of properly maintaining an engine will ensure the technical excellence of their games.

Look at Square Enix and FFVII Remake and Rebirth, both top tier when it comes to production value but with miserable failures on the tech front on thing that from the outside should be easy (like properly loading your textures, properly implementing an upscalling solutions, etc).
I believe in terms of network issues DriveClub was the most bugged.
They had to redo all the net code.

BTW GTS launched with serious net code issues too.
 

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
My final post from what the developer said (on Twitter).

The issue is related to database load.
And it is probably very trick to fix because they already did a vertical upgrade in their database server (they increased from 250k sessions to 360k sessions).

What is a vertical upgrade? Is when you have a server and you increase that server CPU, RAM, I/O, etc… that is very easy on Azure (you have a slide to increase it and pay for it).

But vertical upgrades are very limited.
You can only go up to X CPU cores, Y amount of RAM, etc.

That bring the second option… horizontal upgrades… when you have to add more servers.

And that is hard for some games than others… why some games needs synchronized servers… and that generate a lot of more issues because your net code now is not only dealing with sync with the players but with synced between servers instances.

A recent event of easy game to do a horizontal scaling was Palworld (or how it is named).. it is a game that doesn’t need syncing between servers so they just added several servers each one with a new server instance of the game and called a day… all fixed without need big changes in net code.

But Helldrivers can’t do that… they need to have the servers synced with each other… if they don’t have that syncing between servers implemented in their net code then they are capped until they upgrade their net code to have it (yeap they can’t do horizontal upgrades)… and having to work on net code to sync between servers takes times and generate a lot of new network issues to be tested and fixed… it is a long term option if the net code don’t have that ability yet.

Hey but they can’t do nothing?

Of course they can… but the options takes times to implement these fix.
They can instead (or better in parallel) to plan and work in a horizontal upgrade… work in optimize their own net code (client and server side) in simple terms they can optimize their net code server side to that point to double the amount of players connections in that server without need to do a horizontal upgrade (when you are already at the limit of verticals upgrades).
Or work in your net code server side to take less RAM… so you work to free up resources per session and so you can have way more sessions with the same server capacity.

But again optimizing net code takes time… a lot of time… it is not something you do in a matter of days and that is even more true for smaller teams.
Hell some huge teams took months to rewrite net code with optimizations.
It is a constant work.

That is my view of the issue.

And yea they are facing right now is server side database load issues… nothing can make sure if they find a solution for that issue that others won’t come… right now they are limited by the database load to see others issues.

And that is unrelated with engine of the game on client side.
It is really a server side issue… the server side instance of the game (that is a totally different executable that doesn’t even have 3D rendering capabilities like graphic engines do).

It should be the same using Unity, Unreal Engine, Decimal at the client side.

I hope they find ways to fix as fast as possible but I know how trick it is to fix net code issues in server side.
 
Last edited:

balrogboy

Well-known member
21 Jun 2022
346
347
There are also counter-examples:

Fallout, Starfield, Halo
Dude I'm sure that in better hands the Creation engine could do some awesome shit. Skyrim is better than any fallout game and starfield, not only because the scope fits and pushes the limits of what the engine is designed for, but mostly because its a 2011 game and bethesda is fucking retarded for doign fucking anything else with it after that.

I think that you really can make great stuff any engine, as long as you understand the limitations of said engine. Starfield is the perfect example of that, yes a lot of issues are from a design perspective and should have been scrapped in pre production, but the infinite ammount of bugs, glitchs crashes, loading screens and a lot of the overall design of the game is incredibly limited because they insist on using the objectivelly worst engine they possibly could. I swear to god even unity would be better.

Also dont be a fanboy, Halo is awesome, I say that as a full playstation gamer, even Infinite which was REALLY, like, super fucking lackluster is still better than 90% of other fps games.
 

Darth Vader

I find your lack of faith disturbing
Founder
20 Jun 2022
7,365
10,933
My final post from what the developer said (on Twitter).

The issue is related to database load.
And it is probably very trick to fix because they already did a vertical upgrade in their database server (they increased from 250k sessions to 360k sessions).

What is a vertical upgrade? Is when you have a server and you increase that server CPU, RAM, I/O, etc… that is very easy on Azure (you have a slide to increase it and pay for it).

But vertical upgrades are very limited.
You can only go up to X CPU cores, Y amount of RAM, etc.

That bring the second option… horizontal upgrades… when you have to add more servers.

And that is hard for some games than others… why some games needs synchronized servers… and that generate a lot of more issues because your net code now is not only dealing with sync with the players but with synced between servers instances.

A recent event of easy game to do a horizontal scaling was Palworld (or how it is named).. it is a game that doesn’t need syncing between servers so they just added several servers each one with a new server instance of the game and called a day… all fixed without need big changes in net code.

But Helldrivers can’t do that… they need to have the servers synced with each other… if they don’t have that syncing between servers implemented in their net code then they are capped until they upgrade their net code to have it (yeap they can’t do horizontal upgrades)… and having to work on net code to sync between servers takes times and generate a lot of new network issues to be tested and fixed… it is a long term option if the net code don’t have that ability yet.

Hey but they can’t do nothing?

Of course they can… but the options takes times to implement these fix.
They can instead to plan and work in a horizontal upgrade… work in optimize their own net code (client and server side) in simple terms they can optimize their net code server side to that point to double the amount of players connections in that server without need to do a vertical upgrade (when you are already at the limit of verticals upgrades).
Or work in your net code server side to take less RAM… so you work to free up resources per session and so you can have way more sessions with the same server capacity.

But again optimizing net code takes time… a lot of time… it is not something you do in a matter of days and that is even more true for smaller teams.
Hell some huge teams took months to rewrite net code with optimizations.
It is a constant work.

That is my view of the issue.

And yea they are facing right now is server side database load issues… nothing can make sure if they find a solution for that issue that others won’t come… right now they are limited by the database load to see others issues.

And that is unrelated with engine of the game on client side.
It is really a server side issue… the server side instance of the game (that is a totally different executable that doesn’t even have 3D rendering capabilities like graphic engines do).

It should be the same using Unity, Unreal Engine, Decimal at the client side.

I hope they find ways to fix as fast as possible but I know how trick it is to fix net code issues in server side.
Kinda makes sense to me, you seem to know more about this shit than me as well.
 
  • Like
Reactions: balrogboy

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
Kinda makes sense to me, you seem to know more about this shit than me as well.
To be fair I'm only doing guessing from what they said on twitter.
They shared very little.

But it really points to they don't having the ability to do horizontal upgrades yet (add more servers)... so their options are limited and most like long term options... in parallel works in way to add servers instances (horizontal upgrades) and in optimize/fix your server side net code.

Only the dev team knows exactly what the issues are and the options they have to fix it.
I can only wish the best for them and that they can pull out something fast as possible.

But I know they are a small team.

PS. People say... Sony can send people to help... yes they can but remember that new people will have first to understand the issue and everything the net code is doing to start to give advice to fix the issues... that takes a lot of times too... it is not something you do in a day for example... it is hard to a new team to work and give advice in something they didn't create... they needs first to understand what was created.
 

BLAZE

Active member
20 Feb 2024
206
183
It may seem radical, but Sony should force all its studios to use the Decima Engine, I don't see anything negative about that
 

ethomaz

Rebolation!
21 Jun 2022
11,868
9,662
Brasil 🇧🇷
PSN ID
ethomaz
It may seem radical, but Sony should force all its studios to use the Decima Engine, I don't see anything negative about that
IMO I think the own Naughty Dog engine better than Decima 🤷‍♂️

Santa Monica’s engine too.

And people don’t give enough credits but what Polyphonic Digital do in terms of lighting is something out of this planet and not replicated by any other engine.

Yeap I think Sony studios should continue with their own stuffs… using the same engine should be very negative to their own development.
 
Last edited:
  • Like
Reactions: Gamernyc78

BLAZE

Active member
20 Feb 2024
206
183
Hmmm idk some have their own great engines tht aren't decima.
Santa Monica, Naughty Dog and Polyphony I think are fine with what they use, but the rest of the studios would benefit a lot from just using Guerrilla's engine
 
  • Like
Reactions: Bo_Hazem

Alabtrosmyster

Veteran
26 Jun 2022
3,394
2,935
I never said Azure was a guarantee of performance, but people were blaming Azure for not being able to support the amount of users they had concurrently, which we both know is bullshit. They have much bigger clients and can cope with much bigger volumes, so that is just fanboy speak.

As for bespoke code, exactly, but your engine needs to be able to integrate that bespoke code. So here either the engine can't, that bespoke code is not scalable, or a combination of both. It can absolutely be an engine issue.
That's the point, we don't know what the bottleneck is, they even said they were in talk with Steam... which of course can handle this kind of work load.
 
OP
OP
Gamernyc78

Gamernyc78

MuscleMod
28 Jun 2022
20,386
16,652
IMO I think the own Naughty Dog engine better than Decima 🤷‍♂️

Santa Monica’s engine too.

And people don’t give enough credits but what Polyphonic Digital do in terms of lighting is something out of this planet and not replicated by any other engine.

Yeap I think Sony studios should continue with their own stuffs… using the same engine should be very negative to their own development.
Exactly, there are some top line Sony engines already besides Decima.
 
  • Like
Reactions: Bo_Hazem