The number of players on the server has a lot less impact in client performance than one would think. During the final stages of PTU we ran tests with 50 players, 40 players and 30 players per server. While there was a slight improvement in performance it wasn’t proportional to the player count.
[…] Fill up a Caterpillar with cargo, blow it up over an Outpost on a moon and you can bring the clients and servers to their knees […] Have a bunch of people fly around in Starfarers and Caterpillars and you’re straining the clients and server far more than you would be with a bunch of Auroras and Hornets.
We have solutions for all these things […] but they are not something we can complete in a week or two.
At Citizen Con we announced that we are moving to a quarterly release schedule that is less feature bound and more focused on regular updates […]
If you are getting performance in the sub 10-15 FPS range there is definitely something not right, especially if you have a quad core CPU, 4GB video card and at least 16GB.[…] so please be patient.
Finally I want to say thank you to everyone out there for supporting Star Citizen, your enthusiasm and dedication really does energize the team and myself. We are building something truly special that is only possible because of you.
Happy Holidays everyone!
Chris has made a rare appearance on his weird discord knock-off in order to provide the base with some fresh talking points for the new year.
We don’t really understand why the game behaves how it does.
FPS goes up. FPS goes down. Nobody can explain that.
Faith restored.
I for one appreciate Chris’ astonishingly frank step-by-step instructions to Goons informing us exactly how to take these servers down.
CR’s little holiday diatribe is nice because it reveals his misunderstanding of what’s wrong. His entire post was all about performance this, FPS that, and so on.
As if the performance itself was a problem and it needs to be handled.
Not the case Crobbler - the abysmal performance is a symptom.
It’s like having a kid coughing up blood and phlegm all over the floor, so you say “It’s ok, I’ve got more people coming in with mops, they’ll be able to keep up and keep this floor clean.” You’re rather missing the point.
3.0 is irrelevant - it’s just another inevitable consequence of the mistake they made in 2015 when they released what they called “2.0” - a cynical ploy to prove a “game” existed and an excuse to begin denying refunds. The cancer released to the PU two years ago is simply continuing to advance through its stages.
We’re at stage IV now.
It’s good to see that they’re giving some consideration to the idea of shifting away from a model that stream all data everywhere at once to the player regardless of location.
It’s an unconventional approach but this is an unprecedented undertaking.
Chris’s Christmas message:
The game is shit. We don’t know what’s going on. The ships and cargo are fucking up the engine.
We might address things mid January, or later.
Merry Christmas.
From the data we see it is not so much about player count but more about WHAT the players are doing. In our internal testing we didn’t witness the performance issues that we saw on PTU or Live once thousands of players got in and started doing all sorts of crazy things.
Crazy things? You mean crazy things, like, oh, I don’t know, trying to play a sandbox game like a sandbox game, which was the expectations that you set from Star Citizen’s very beginning?
You should see plainly now, Citizens: Croberts is already the first to blame you, the esteemed pledgers, when anything starts to go wrong with his game.
CIG has what, 15-30% of their ships flyable, I’m guessing >10% of the player items/armor they envision for the full release. They have no idea why performance is garbage with just this small set of assets. MoMA help me out; what fantasy land do you reside in that this starts working and more than 10 players can engage in complex twitch combat? Have you already admitted to yourself that this project will never see a 30 person battle?
Absolutely none of this is built in a modular or scalable fashion. They’ve said repeatedly that every ship is a special case and this is also evidenced by years implementing the small handful they have currently.
I guess we’ve finally reached the final stage. Chris never had a vision, and nobody else at CIG knows what Star Citizen is. Their plan appears to be to take a bunch of assets and dump them in a box:
And somehow “emergent gameplay” will spring to life from that. But nobody can get any of the assets working, or the physics, or anything at all.
Half a decade, $170 million, and a blithering idiot who willfully misrepresented his ability to lead this project.
And now, just in time for Christmas, a new Articles of Faith is released to keep the backers pumping him his unearned and undeserved windfall so he can continue to fail at every single thing he attempts.
Every day is Peak CIG.
Derek Smart
I knew croberts had balls. But I had no idea they were Platinum. I just read his latest. Wow. Not only is he blaming gamers for doing precisely the features he sold them, but also managed to blame them for the server issues.
So Crobbers is saying the players are doing weird things, like starting the game.
To be fair he’s not wrong
Let’s flashback to when CIG was first rolling out their cash shop and people (before the narrative was given enough time to shift) reacted very poorly to the system that remains in place to this day.
That spectrum thread is great, backers detail just how broken the game is, Chris and Toni throwing nonsense words at them until they are appeased, backers then fawning over them for taking the time to do “work” on Christmas. Star Citizen is good.
Derek Smart
I spent 15 mins in that thread. I didn’t know what to expect. Then I realized that, for as much as we take this to be a joke, this - this - is a fucking cult. Can you imagine what would happen if EA, Activision et al pulled this shit?
WE’RE SANDY AND CROBERTS, AVARICE AND GREED
WE TOOK ADVANTAGE OF THE WHALES, JUST IGNOOORED THE GLA
WE SPECIALISED IN JPG SALES, SPREADING DREAMS AND SCOPE
AND IF YOU SPAWNED THE WRONG TEST SHIP
WE SIMPLY THREW YOU OUT
So It’ll take them till june/july to add the content that was originally schedule for 3.0 back into 3.0. So that leaves them 6 months to add mining into the game when it took them a full 2 years to go from video presentation of 3.0 to actual 3.0 release.
They are complaining about cargo ships breaking the game, imagine when mineral haulers get added to the game. Imagine when cruise liners need to be simulated into the game. Imagine when the cruise liner gets blown up as its arriving into port and the server has to simulate all those corpses
Just a reminder on what 3.0 promised vs delivered:
CR: I would like to point out that ever since I got the legal threat from Skelling I have been working on this response. I worked on this until 5am last night, and a couple more hours this morning in the UK, where I am currently am in preparation for ATV in a week from Saturday. Conservatively its taken me about eight hours to write. This is time I could have spent working on the game instead of dealing with a Crytek instigated drama. And this is really what annoys me that their silly lawsuits occasionally gain traction and pull me away from the very thing I prefer to do and the very thing everyone wants me to do and the very thing Crytek accuses me of not doing CHANGING THE ENGINE/NOT CHANGING THE ENGINE (-note -check with Ortwin which one I should go with)! By constantly nagging, writing warning letters and soliciting solicitors in the background to enforce a dumb old licensing agreement, Crytek is waging guerrilla warfare on my time, the time of other key executives, and the peace of mind of our employees and backers.
I have to believe that somewhere in CIG high command, somebody has come to the realization that the mmo is never going to happen without an enormous, very lengthy total rewrite. What they built so far may be good for Sq42 drop-in/drop-out coop with like 4 friends, but that’s about it.
I think, because I have to think, is that this keeps getting reported to Roberts, but he either ignores it, or demands people soldier through the problem because he came up with the solution already (mesh tech) and they just need to apply the code to it.
I believe that CR sees “mmo rewrite: 6 years to alpha” and he can’t wrap his mind around another 6+ years just to get a very basic functional vertical slice of his dream mmo out the door. He sees the arena game, and sees how it works, but that it just needs more tweaking with his great ideas to make it a 10,000+ player mmo with his magical ideas.
Now they are basically where they were 1.5 years ago with an engine that still can’t work, except now the added in hack jobs are really starting to add up. The system literally can be extended no further, and even the best computers can no longer play it.
Lack of planning, lack of comprehension, and hubris are going to force more people away this year since the only thing they can do to make this mess work is rebuild it as an actual mmo, which will take a decade or so, or remove components to help get it sort of functional again. They may be able to figure out how to band aid stuff, but they’re looking at teeny tiny gains out of that.
The system literally can be extended no further, and even the best computers can no longer play it.
I’m gonna zero in on this point because it contains something that bugs the shit out of me. This insistence from Chris Roberts, and the cultists, that you need a “powerful PC” to play his game, the implicit statement being that his game is an avatar of gaming perfection that requires more than a mere mortal vessel to contain it.
The reason it bothers me is because for years I’ve been fighting one of the most pervasive lies in PC optimization history regarding the need of a “powerful PC” and why that might be the case. There are precisely two, and only two, reasons you need an absurdly powerful PC to play a game (or do anything else):
1) The game is using new graphics technologies that are just now in their infancy and might not even be optimized or build into the GPU architecture, they’re pushing the edges of realistic physics calculations, they’re pushing the edges of how many on-screen players can be in a single battle - anything where the developers are 100% aware that they’re pushing it, and usually realize that in a couple of years hardware will be more than capable of handling it. “Max Settings” for these games is usually nothing more than hedging their bets.
2) The game is complete shit that fails to optimize nearly anything, wastes CPU cycles on nonsense that doesn’t affect the player, overheated GPU contributions that are not visible to the player, constant calculation of irrelevant AI, and just wastes I/O through poor decisions and logic.
Star Citizen is #2 masquerading as #1. Star Citizen is not doing anything that hasn’t been done before, far better, by smaller teams, and for cheaper. But there’s an undercurrent to their bullshit that they’re the next Crysis (engine choice deepens this illusion) and they’re pushing the envelope so hard you guys that your PC running Star Citizen well is the de facto test for whether or not your gaming testicles have dropped.
It’s all bullshit. It runs like shit because it is shit, and the only thing better hardware will do is allow Star Citizen to crash faster.
Even if Star Citizen had the best technical and managerial leadership in the world, it still could not achieve what they want with the current generation of engines. The ambition and scale they boast so much about does not “merely” require skillfully adapting current technology. It requires a paradigm shift applied to an entirely new engine.
Nevertheless, with the massive budget they had, there was a chance to make the best of the engine they were contractually obligated to use. This would involve doing the proper analysis and protoworking. Working out what the challenges were, taking the time and effort to decide where to compromise, where to develop entirely new technology and where to experiment. And not actively building a playable game while doing this.
Instead they’ve simply layered every single thing on top of the existing systems. I am hard pressed to see an instance of actual innovation and novel development. Everything is CryEngine hammered into shape until it fits or breaks - and then duct-taped or covered up, if it broke. I am baffled by this approach. Why didn’t they build actual server technology? Why is there no custom network protocol? Why didn’t they build a custom system to handle scalability of physics? Every single issue they’re having to solve now just to make 3.0 playable, could’ve addressed at the beginning of the project. Even with the vague and ever-expanding scope, they knew they were build an MMO with spaceships.
I wasn’t along for this bizarro ride early on, as I took one look at the original pitch, was tempted but then remembered how difficult it is to build a space MMO, and decided to keep my wallet closed until they had more to show and I was confident this was not going to be a shallow pay-to-win game. But can someone who was a backer and fan back then comment on why early development took this weird path of refusing to make any new technology, and instead simply pretend CryEngine could be hammered into an MMO engine?
Pleadings, rulings, motions, etc., are public information and will be available to people with PACER accounts a day after it’s filed with the court. The response to the complaint can be more or less a single page with Ortwin scribbling in crayon “lol no” to all the allegations and that ought to suffice for the purposes of entering a response and avoiding a default. Of course the proper way to do this is to lay an appropriate foundation for your defense, and there are certain things you must plead at this time or it’s waived (jurisdictional flaws, e.g.) but CIG can clownshow the case for a little while longer with the bare minimum needed.
Given that low, low barrier I don’t expect a default but…it’s CIG. Who the fuck knows. Maybe this will explode in a supernova of stupidity before it finally fades away, the bright star in the sunk cost galaxy that shines all too briefly before being devoured by the Skadden supermassive black hole.
A friend who would like to stay anonymous felt compelled to write an effortpost about the performance situation in star citizen 3.0 and CIGs future outlook.
It’s s a long post but it’s well worth the read.
I take no credit for this.
Performance, i.: Where is the PC Master Race?
A core tenant of Star Citizen’s advertising was appealing to this facet of the PC master race, remaining unshackled by the plebeian hardware of consoles and leaving the sky as the limit for the smooth, buttery goodness of a game that utilized more fully the current - and future - maximum potential of gaming PC hardware.
And yet Star Citizen is currently a slideshow that frequently dips into the single digits. This is unacceptable for even a console title, of which there are many TCRs (Technical Certification Requirements, i.e., “the console maker won’t let you ship your game on it if it fails these”) regarding performance. That is why, for all of Reddit’s angst, you very rarely see a console game that actually has noticeably poor performance. As has been consistently demonstrated over the past decades, 30 FPS is simply Good Enough.
But Star Citizen doesn’t even have that. And it is unlikely to have that in the near term. And it is unlikely to remain sustainable as ever more features are bolted on. Performance is, ultimately, one of the final litmus tests by which a game’s programming is judged. It is the sum total of all of its parts, working in concert, in real time, on a player’s hardware. Performance is unlike any other aspect of game development, in that it can almost never be simply patched after the fact. It must be explicitly designed for from the start, as no matter how many people you assign to the task of “performance is bad,” eventually, they stop being able to continue to claw back the framerate in 1 and 2 FPS increments through additional work.
Why is this? Join me for a guided journey of the dark and mystical land of low-level performance optimization in video games; a glimpse behind the curtain of what outwardly appears to be nothing short of wizardry with extra steps.
Performance, ii.: Design philosophy.
Every project is different, as is the way that performance impacts it. Some projects never need to worry about performance, such as the common meme of the student’s simple Python script that processes some small amount of data, perhaps putting it into an excel spreadsheet, which is sufficiently small that it simply doesn’t matter how poorly optimized it is. So long as the data is good, no one cares if it takes one second or three seconds to execute.
Some projects, such as games, live and die by performance, as no one will play a game that is a literal slideshow. Therefore, the game’s architecture must be designed properly from the very beginning, as perhaps a mere 25% of performance optimization is the result of clever programming tricks, with the remaining 75% being the result of good design.
The house analogy has constantly been used to describe this phenomenon. And with good reason, as it is quite accurate. The design of code is tricky business, as it is something invisible to players. Anyone can see if the jackets have enough fidelity or not, and anyone can see whether the fluid dynamics in the drink glass provide sufficient immersion for the eventual drink mixing minigame. Conversely, only a tiny fraction of individuals, who are both highly technical and highly experienced, will be able to see whether or not code design is proper.
“But it’s an alphaaaaaa,” come the cries of the faithful. No. The entire point of this is to expound upon the fact that these issues are ones which should never be encountered by the time a game gets into the alpha stage. These are things that must be executed properly before many of the game’s systems are in place.
And by all accounts, the people at CIG who were experienced enough to properly guide the design left long ago, leaving what seem to be primarily inexperienced members to attempt to do so. I have nothing bad to say about those working in the trenches, as it is not their fault at all. If someone gets told to implement volumetric fog, and they do so, then that should in fact be celebrated - it is the management who decided to mis-allocate the resources to prioritize volumetric fog over more important tasks which should earn one’s ire. Similarly, I have nothing but respect for those who still attempt to fix and improve the code’s design even if they are unqualified to do so - I trust that they are in fact trying their best. These criticisms should be, once more, directed to those who mismanaged the project and put them in that position to begin with.
Performance, part iii.: Case study of poor design.
“One issue is with animation updates of attachments, specifically doors.”
Doors!
Doors are a prime example of something which has been needlessly over-engineered. Before, I have heard stories of ridiculous things such as fixing an issue with moving through doorways by placing invisible physics objects inside of them rather than properly fixing their collision bounds. While not directly related to performance, this is the kind of thing that we call a ‘kludge’ - a quick and dirty fix that is not considered proper. It’s like fixing a rotting stairway by covering it up with pieces of plywood. It might work in the short term and allow you to continue traversing the stairway with less risk of putting your foot through a weak piece of wood as you take a step, but it is certainly not a proper solution. Because next month, you might want to add a ramp to your stairway to allow access to those in wheelchairs, but now you not only have to work around the rotting steps, but the plywood you dumped over it to cover it up. Similiarly, the next time they want to change how doors work in game - perhaps they need a new type of sliding door, for example - they now have to work around the kludge they left in there from last time, assuming that anyone even remembers it in the first place.
The fact that there are evidently performance implications regarding doors should therefore be of little surprise to anyone.
“Moreover, we have a big issue with badly spawned ships for interdiction that causes significant but unnecessary physics that delays the execution of the next frame on the main threads (and thus severely impacts frame rate).”
Here we go, a nice meaty subject. There are two parts to this:
1) Objects with physics that we don’t need.
2) Multithreading delays.
The first issue is quite clear cut: due to the eldritch horrors of their frankencodebase, they’re performing physics calculations on objects that do not need them. This is indicative of those systemic issues I’ve just described, as this sort of thing should be very, very easy to trace back to where it is occurring. Since it’s not, as it should theoretically be a five minute fix in a properly designed engine, their ship spawning system can be assumed to be a rats nest of different things interacting rather than one neat, orderly, centralized thing which handles all of the necessary spawning logic.
The second simply means that the physics thread has too much work to do, which is causing the game to run slowly. It sounds simple - “hey, we just need to do less physics work, and then we’ll be safely back in the 120+ FPS PC Master Race, right?!”. Sadly, it is not. In fact, it describes something really, really bad for Star Citizen.
In most game engines, physics (all those math-y calculations that the computer uses to figure out whether your commando is properly standing on top of a ramp or T-posing his way into the great unknown, for example) and rendering (drawing all of those fidelitious layers of jackets, for example) happen separately. This is very important to maintaining PC Master Race membership, because it means that if too many physics things happen at once, the framerate will not drop. In a properly designed engine, you could have ten thousand commandos running into each other at once, and the game would still remain at the same framerate - the simulation would simply slow down, and everything would move in slow motion.
An easy example of this is EVE Online - there can be huge engagements that crawl along, with actions that normally take seconds taking hours, but during it all, you can still move through the game at a high framerate. This is because the simulation and rendering are decoupled, and the simulation taking longer does not slow down the rendering.
In SC, as they just described, this is not the case. This is Very, Very, Bad. If EVE Online were designed in the same way, then those enormous engagements would be completely unplayable, as the game could only render a new frame after it was done doing the simulation calculations that took an hour to do. In the meantime, the game would appear to hang, and be completely unresponsive - for all you know, it crashed. This is exactly what is currently going on in SC, just at a much smaller scale. The design is neither scalable nor sustainable, and is a really bad thing as it only works if the physics will never, ever be doing enough that it slows down rendering. Assuming that networking works perfectly tomorrow, how exactly are those capital ship battles going to play out, with this limitation in mind?
“Moreover, the team’s going to build a thread local queue to push physics commands in order to avoid synchronization cost caused by the large amount of physics parameter update calls (in order to ultimately get down to reasonable numbers again).”
This is the sentence that actually prompted me to begin writing this series, as it struck fear into my very soul.
The system they describe is fundamentally changing the manner in which commands are issued to other objects in code - in this case, related to physics.
Most code written for games is directly invoked:
Object.DoThing();
Object Does the Thing as soon as this is called. Pretty simple. But let’s say that you need to Do A Thing on a different thread, such as physics. You now need to worry about how to properly synchronize data between those different threads, because in any multithreaded system, code running on one thread does not know about anything running on a different thread unless you specifically and carefully make them aware of each other.
One way of doing this is to block both threads. This works, and is easy to implement:
BlockPhysicsThread();
Object.DoPhysicsThing();
UnblockPhysicsThread();
Remember Spinlock? It’s the same thing. And remember how it tanks performance? Yeah, same thing.
A queue is another programming construct which attempts to solve this issue. Internally, this creates a little note that says “hey, mr. physics thread sir, next time you get a chance, could you Do This Thing? kthxbai”. Simple and elegant, and how many games handle this type of situation.
If this system was properly designed in the first place, it would be no big deal at all - all other code would have been written with that in mind. But now, they have to go back and change it in order to prevent even more bugs - or worse, outright crashes - from being created as a result.
“In addition to that, the engineers are trying to reduce the amount of items / entities / components and their updates that contribute to the high baseline cost on the client.”
This is the same as previously covered - things are being updated that don’t need to be. Let’s explore that issue on a deeper level. A performance optimization that I have heard much about of late is distance-based culling - which is, quite simply, “don’t update things that are too far away.” That sounds pretty simple, right? It is. In fact, it should theoretically take about two lines of code:
if(Distance(Player, OtherThing) < WayOverThere) {
UpdateOtherThing();
}
Or at least, it should be that simple if the engine was designed for it in the first place. Since 1) it is clearly taking them months and months to do this instead of about twenty seconds, and 2) this technique is one of the simplest and most cost-effective optimizations that almost every game ever written ends up using it in some way or another, we can surmise that there is something causing a problem. What could that be?
There are a multitude of reasons as to why you can’t simply bolt this on after the fact. Many of them stem from the same problem as in multithreading: there’s a lot of other code that expects the code you’re trying to optimize to behave one way, and changing its behavior as part of a performance optimization can introduce more bugs and crashes, or even become impossible without also changing all of the other code in some way.
Maybe you have a seemingly innocent bit of code tucked away somewhere that checks something innocuous, such as whether a chariot’s lights are turned off or on so the renderer knows whether it needs to prepare some extra lighting calculations for them or not. Maybe that piece of code was hastily thrown in every ship’s Update() function, because some guy came barging into your office and demanded that the lights look extra fidelitious in the next check-in, and you wanted to go home because it was already 11 PM and sleep started to look more appealing than bulletproof code design. We’ve all been there. Maybe this was lost in the shuffle, hidden in a tiny note in your source control commit message that you promised yourself you would fix later. Maybe later never came because someone barged into your office the next morning and insisted that you work on important logic for AI commandos to recognize mopping as their highest purpose in life.
And now, months later, someone else is desperately trying to improve performance at 10:30 PM for a big demo the next day, and tries to add in culling logic that stops calling Update() on ships that are far away. And when he compiles and runs this, he now finds that the lighting is wrong on all of the chariot’s lights. Stymied, he now has to trace through all of the code that could potentially impact that. If he’s lucky enough to eventually find out that tucked away UpdateStupidLightsSoICanGoHomePlz() function in a bad place, now he has to decide how to fix it. Does he properly refactor the light checking to work on the renderer’s side, staying until 1 AM? Or does he pull that bit of code out and plop it into the messaging system he’s been writing all day, kicking the can ever further down the road just so that he can get home at a reasonable hour such as midnight? Decisions, decisions…
**This is very complex, abstract stuff, and even professionals have trouble getting it right on the first try. Software is somewhat unique in that respect, as in most other fields involving complex technical endeavors, such as building a huge skyscraper, you have the benefit of being able to touch and see it. Building a tower makes logical sense. We instinctively know how gravity works, and that a pyramid is a very strong structure and an upside-down pyramid is probably not.
But in software, we do not have that luxury. It’s all abstract concepts. And while we can make fancy flow charts attempting to show relationships between pieces of code, it can often feel like the flatland problem - that we’re mere 2D beings helplessly flailing about, trying to visualize 3D concepts that our poor brains simply weren’t designed to handle. Software development is all about taking those abstract concepts and solidifying them into functional code.
And that code is, by design, very brittle. If everyone agrees that we’re making a pyramid, and then halfway through stacking blocks to build it, someone waves their hands and insists that we’re actually making an upside-down pyramid, it’s probably going to fall over. And when software is poorly designed, on any level, that is exactly what happens.**
Performance optimization has the worst of all worlds because it is bound by hard technical limitations. Time and money to add new features may indeed be infinite, but at the end of the day, the CPU cycles on the player’s computer are very much finite.
‘Twas the night before Paarpmas, and all through the ‘Verse,
Not a backer was pledging, for it just got worse
Delusions were posted on Reddit with care,
In hopes that St. Christopher soon would be there;
he showed with an air
