T O P

An in-depth discussion of the pop system, its impact on performance, and possible solutions.

An in-depth discussion of the pop system, its impact on performance, and possible solutions.

maqianli66

Actually every pop does more calculation every month than you think. Aside from what you already said, each pop also calculate their individual resource consumption, hapiness, ethic and habitability. As for your proposal, there is actually [a mod kinda taking in that direction](https://steamcommunity.com/sharedfiles/filedetails/?id=2409578462). It's not yet updated to 3.0 though, and since the author has a lot of mods to maintain, it would probably take a few weeks for the mod to be updated.


Nighthunter007

That looks like a very interesting project, essentially skirting around the limitations I imagined to do a lot more than I expected was possible. I'll have to take a look through the files on that. Fair point that they do a lot more as well; happiness, consumption, ethic drift, automatic resettlement etc are all O(n) operations, and would in this suggestion be O(s) at most (some of them would be O(1) if they don't depend on species type like ethic). I focused on job selection as that's the headline feature when it comes to explaining the slowdown (though you'd think in 3300 words I could have fit this sentence too).


NobbynobLittlun

> Actually every pop does more calculation every month than you think. Aside from what you already said, each pop also calculate their individual resource consumption, hapiness, ethic and habitability. That's not an issue. Those are just more "features", as they are called in data science. Another row/column in your linear algebra transformation. Although, I think I'd try to instead figure out how to approximate it as a series of 4-dimensional transformations, so that you can pipe it through the graphics card.


NotATroll71106

Those other things should have a negligible performance impact unless Paradox has really shoddy code. Consumption, happiness, and habitability are just simple arithmetic that should have the same execution time per pop no matter how many there are. Ethics should also be pretty simple based on the factors I've seen in the wiki. Granted, I could see how a coder could pick an inefficient method for determining factors like "A Repugnant alien pop is on the world". I could probably write code that would handle a million pops doing those calculations every second.


BlueCommieSpehsFish

At this point I honestly believe Paradox just really does have very shoddy code.


HAK_HAK_HAK

I thematicallly like the individual pop system, as it feels a little less "spreadsheet-y," but "spreadsheet-y" is preferable IMO to destroying the RP of having a huge empire.


Nighthunter007

I agree; thematically the pop system is nice (although I don't think I've ever actually looked at individual pop happiness when I have hundreds of them). I tried to keep a lot of the spirit of the pop system, putting as much of the changes as possible behind the scenes. You do lose the ability to see the individual portraits of the pops working each individual job, but overall I tried to keep abstraction to a minimum.


nelliott13

You wouldn't even need to lose the portraits -- I think it would be easy for the interface to display one portrait per pop working a given job rather than as a pie chart. **EDIT: see explanation below, this is not how the proposed process works!** This would be abstracted still in that moving pops from one planet to another wouldn't (necessarily!) move the same portrait and that individual pop happiness and ethics would not be assigned. But individual pop happiness and ethics are not really relevant to current gameplay anyways, since ethics and pop happiness affect planet-wide happiness and output via stability, not individual pop output. Plus you could assign happiness and ethics to individual portraits via some sort of normal distribution around the planet/species averages if you really wanted to.


Nighthunter007

Species portraits on jobs would be lying by necessity – though that might not be a deal breaker at all. If you notice, I don't actually calculate how many of each *species* are specifically assigned to each job. You could, but then your runtime would probably increase to O(sk) rather than O(s+k). I've sort of "cheated" past that (which I'm not sure hasn't introduces some double-counting of bonuses, incidentally), but an O(sk) solution is still independent of pop number so it'd probably be fine. Cheating a bit with the portraits might be fine though. The Democracy series does something like this, actually. It's all (as far as I know) stats-based, but you can bring up a sort of "focus group" with a representative set of attributes and level of support.


nelliott13

Hmmm, I didn't catch that. Am I correct in thinking that a pop's species (or more accurately, its traits) influence job selection is step 2? **EDIT: This thought was wrong; see explanation below.** If not, how would the specialization you discuss work? If so, how do you link the trait information of step 3 to the pop distribution in step 2 to calculate the final job output in step 4? I'd think that information has to be stored somehow in a way that could be summarized at the species level? I'm envisioning this in terms of matrix multiplication: * Step 1: a one-dimensional array of pop count by species modified in step 1 (s rows, 1 column) * Step 2: a two-dimensional array of species pop per job (s rows, k columns) * Step 3: a two-dimensional array of species bonus per job (s rows, k columns) * Step 4: a two-dimensional array of base output per job (s rows, k columns) that gets multiplied with the prior matrices to get final output. And the arrays of steps 3 and 4 would be relatively static and could be combined into a single two-dimensional array of species output per job (s rows, k columns) to save a monthly step. In the absence of the information being easily available or summarized, a focus group seems like a good workaround, as it would only need to be calculated on a planet for which you open the jobs tab.


Nighthunter007

It does not influence job filling in step 2. Step 2 is just about seeing how many, say, Clerks we will have. Then, in step 3, we look at the number of (eligible) pops with the Thrifty trait, and use that (with specialisation) to calculate a sort of "average Trade Value bonus from traits" for essentially the whole population. Now, I may just have counted every *trait* rather than every *species*, which could be a problem. If I have a species that is, say, Industrious and Agrarian it will actually count towards both. At 100 specialisation, every pop will confer its bonus to both professions. We could probably overcome this by calculating this in tandem and essentially either implicitly assume an even proportional distribution between every job with a matching positive trait or hard coding some factors for each trait (so Intelligent might be more important than Agrarian). You take the total number of filled jobs that this species has a bonus to. They can be filled with "any species" in a way; in code they would just be "filled". Say 100 miners and 50 farmers and a species with Agrarian and Industrious. Then take the e.g 90 pops of that species. 1/3 of their bonus would go to farmers and 2/3 would go to miners (though this can be weighted). Then you go 15%\*60/100=9% and that's your bonus to farming. Do the same for every species, to a maximum of 15% (when all pops would be Agrarian). With a specialisation of less than 1 this decreases linearly towards just pop_with_agrarian/pops, which are both really simple numbers. In a way this does implicitly assign pops to jobs, but only partially. If we had 500 of that species we wouldn't care about the other 350 after we've calculated the bonus to farming and mining. This way the step is O(s\*number_of_resource_traits) rather than O(s\*k). This is, however, a difference of only ever a factor of 5 or so. I did it this way to avoid any O(s\*k) operations in order to keep my runtime O(s+k). However, s\*k is still independent of the actual number of pops, so would also solve the issue. Your method is probably a little cleaner and might even be faster in practice if it allows better memoisation (also matrix multiplication is blazing fast in a lot of frameworks), not to mention it actually allows showing portraits properly. It might make specialisation harder, but that's not actually a current feature of the game.


nelliott13

Ah, that makes sense. That's an elegant way around choosing a job based on traits -- thanks for explaining! Yeah, who knows how fast the matrix operations would be. Definitely more calculations than your approach, especially in creating the species x job matrix in any way other than an even distribution. There may be some clever way to calculate a pop's comparative advantage in matrix form?


Nighthunter007

Honestly, it might not be that many fewer. A species might have 3 different jobs available that match its traits somehow, out of the 12-ish total jobs. So it's a factor of 4 in that one loop. Laying out memory correctly is much more important at that point in practice. A well-built matrix multiplication should be very predictable (and therefore give few cache misses), but that's getting well into the weeds. I know almost for a fact that your method would be faster in Python (because loops are slow in Python), but that's not really super relevant is it.


SkillusEclasiusII

Your system would definitely be more efficient, but I wanna do some musings on the current system as well. Some observations, hypotheses and ideas, if you will. Disclaimer: I don't know the code either, so what I'm saying here is not necessarily based on facts. There might be things I'm missing. To start, monthly job calculations really shouldn't be a thing. If nothing's changed since the last month, there is no reason to rerun the calculation. Jobs should only be calculated when a new pop grows, you resettle, a building finishes, those kinds of things. On the one hand, I would expect this to already be the case. Paradox does stupid things a lot, but they're not idiots... right? On top of that, you wouldn't need to recalculate all pops. First do the one that is new, then do the one that just got kicked out of their job, and so on. Of course, if this is done, you do need to ensure that this chain ends. It's also worth checking how many pops this will generally recalculate. If pops are highly specialised, I'd think not that many, but who knows. Maybe the monthly lag spikes come from some other source. Someone in this thread mentions ethics. If pops are individual agents, recalculating ethics on a monthly basis will definitely add calculation time proportional to the number of pops. One thing they could do to reduce calculation time caused by pops is to group the same pops together. If there are ten identical miners, there is no reason to run the same calculation for each of them. Though... I think stellaris probably already does this to some degree. In my experience, end game lag is a lot worse when playing xenophiles and especially xenocompatibility. I'm not entirely sure how much this would affect the normal calculations, but it definitely makes grouping pops much less effective. You can't group pops if they're all unique. Another thing to do to improve performance is to sort pops by efficiency. (Again, they might already do this.) That way you make it far more likely that something closer to the best case scenario occurs. Of course, it wouldn't change the worst case complexity at all. That's all I can come up with right now. Ultimately though, any pop based system will always scale to some degree with the number of pops. So yeah. A statistical system is probably better.


Nighthunter007

You're making good points; all of this is stuff I'd look at if I were a PDS dev working with the current system. I expect several points are implemented, and the ones that aren't probably have a good reason. They've talked in the past about optimising by these kinds of "tricks" or efficiencies. But your last paragraph was kind of the point of the post. Whatever optimisation, ultimately it grows with the pop size. Since I don't know what optimisation is possible or already implemented, I wanted to take a more high level approach and try to remove pops as a performance factor altogether. My proposal certainly has similar optimisations waiting (no need to recalculate a planet if no new pops or jobs were created), but those aren't relevant in the limit (can only really be known with a profiler running).


Dorgamund

Honestly, one thing that I would guess contributes a huge amount is the AI. Like, PDX AI is notorious for having issues, but how much of their functionality in general is tied to pops? Like, if pops are self calculating a bunch of different functions, is that what is actually supporting the AI system?


Nighthunter007

I can't remember if it was for Stellaris (I think it may have been EU4?), but at some point they showed us a tick time graph. Essentially what part of the game took how many ms in each tick. And the AI was pretty up there. A not insignificant part of the slowdown in Stellaris is down to fleet movements, wars, and all the AI decisions that come with those things. The pops are pretty self-supporting, though, that is true. More, at least, than the tile system where you needed to move them around to the right spots.


Aazadan

If I remember right, a big part of the issue with typical Paradox AI is that fungible stacks are incredibly resource intensive to calculate. But, at the same time if the AI can't break a stack up, it tends to result in them never combining damaged fleets, and therefore almost always just having fleets in a state of constant repair. Throw this in with the ability to heavily customize ships in Stellaris and it gets to be something of a nightmare when it comes to trying to handle things efficiently.


DiabloGoreOrRiot

I think they could just apply fuzzy logic to optimize calculations of metrics required for the AI to make decisions. The AI doesn't need exact numbers of all the metrics to make a decision, thus these input parameters can be calculated in much more relaxed way. It can add to unpredictability of AI and make it less consistent, but this can be ignored.


Chansharp

> Paradox does stupid things a lot, but they're not idiots... right? glances at CK2 Greeks constantly checking if they can castrate everyone


ThreeMountaineers

> To start, monthly job calculations really shouldn't be a thing. If nothing's changed since the last month, there is no reason to rerun the calculation. Jobs should only be calculated when a new pop grows, you resettle, a building finishes, those kinds of things. On the one hand, I would expect this to already be the case. Paradox does stupid things a lot, but they're not idiots... right? Didn't they use to run job checks daily when they changed to the current pop system?


BlueCommieSpehsFish

yes


innocii

> First do the one that is new, then do the one that just got kicked out of their job, and so on. You wouldn't even necessarily need to do this. Because the new pop would write applications for open jobs, right? Just check which open job fits best. And if there's none, it is unemployed, tough luck. How about resettling to this juicy new Ecumenopolis with better job chances? This is just like it works in real life as well (mostly).


DrShadowstrike

A big flaw here is that this model does away with ethics, and any modifiers associated with each pop. Your human refugees and long settled humans are now lumped together, as are your authoritarians and spiritualists. There is also no longer any interactions between ethics and jobs. You would have to treat each group with different traits separately to represent that, at which point you might not be saving all that much in terms of computational complexity.


3davideo

I think ethics could be quite easily be implemented in the same way Vicky2 implements ideology - as a pie chart for each bundle of pops. Example: the Human Artisan pop on the Martian Ecumenopolis could have a population of 18357 which is 42% Egalitarian, 28% Xenophile, 16% Materialist, 8% Militarist , and 6% Spiritualist.


gamas

It's worth noting though that Vic2 does an individual pop system though. The pie chart is just calculated based on the individual ideologies of each of the pops. It's just pops in Victoria 2 are depicted in an actual spreadsheet and you can't really influence them directly so you don't notice until your game slows down in the late game. I think its hard to abstract away these concepts without losing the dynamicness. EDIT: Like I'd say what they've done with the pop limits is perfectly fine, its just they overshot from a balance perspective.


Nighthunter007

I didn't mention ethics, but population-level ethics is still doable, even if individual pops will no linger have their own ethic. I don't think there are a whole lot of ethics-based pop-level modifiers, are there? Jobs affect ethics attraction, as do some traits, but that can be done in O(s) time and O(k) time respectively and applied to the population-level ethics attraction and shift. Things like new refugees; you're right they would no longer have an individual effect on the specific job they are assigned to like today. They would sort of "blend in" with the population as a whole, but you can, for instance, lower the aggregate population by 5% * (#refugees/#pops) which has the same effect on the overall happiness as today and is done in O(1) time. It definitely entails gameplay changes (some good, some bad), but so does effectively halving the total population by 2400 which was the solution in 3.0.


DrShadowstrike

I can see two solutions here: 1) Subdivide the larger, variable sized "pop" unit based on combinations of ethics, jobs and event modifiers. That would result in more of the larger "pop" units (especially when you need to add species and strata to the mix), and it's kind of a question of whether you'd end up with fewer pop units or not. 2) Apply some new system of ethics/job distributions for each of the larger pop units. This would reduce the degree of abstraction of how much those things affect individual pops though. You might get some computational efficiencies either way, but it becomes a little harder to disentangle how those different factors affect each subgroup of the population separately. Right now, that's not an issue, but the issue here is that it makes it trickier to build on (and greater representation of the ethics and modifiers on each subgroup of pops is necessarily for a deeper political simulator).


Nighthunter007

I think ethics can be completely separated from the pop – represented only on the aggregate (planet) level. From that I can also glean information such as "my population is 32% militarist, and that's partially because 52% of my population is Strong or whatever. Show the total attraction values (which the total will drift towards, just as now) and show the break down on each ethic for what's causing it. If 52% of my pops are Strong (i.e are from a species that has the Strong trait) then that would apply a Militarist attraction exactly 52% as strong as if all my pops were Strong. The current effects, as far as I can see, go only one way. Pop traits and jobs affect ethics. Governing ethics (but not pop ethics) affect output. Pop=>ethics effects are easily mapped to species=>ethics and jobs=>ethics, governing ethics effects are just applied to the total job output. There's no reason you need an agent-based approach to population in order to simulate politics. Hell, the Democracy series is all about simulating politics and there's not an agent in sight anywhere. I don't think there's a particularly strong reason to associate ethics or happiness with individual pops at all. In fact, this entire excercise is about how to remove any per-pop O(n) calculation. Subdividing pops or storing any pop-specific information would reintroduce calculations in O(n) time, defeating the entire purpose of the suggestion.


DrShadowstrike

I think that's where we're reaching a fundamental disagreement. I think it is important to have pop specific information, beyond simply representing it as percentages. Knowing that a group of humans is 12% artisans and 52% militarist isn't as meaningful as knowing how those intersect, and how that might have differential effects. The current system doesn't do that very well, but that's a clear direction that can be improved on. If you go to the abstract level you propose, then you close the door on that kind of political simulation, which is an important element of having pops in the first place. You might as well do away with these things altogether, and have planets be represented with an "improvement factor" and just contribute net sums of resources.


Nighthunter007

You can still look at the impact from each job on each ethic, like seeing how the soldiers are increasing the overall level of militarist sentiment of the population on a planet. In fact, it would be easier than it is now to figure out what exactly is affecting the makeup of your pops' ethics without wading through 120 tiny overlapping portraits in my metallurgist row to figure out how my various species traits are pushing my ethics. Though, of course, the game could choose to hide such detailed specifics if it wants to be more in line with today's version. You could also emulate a lot of the data you want by just looking at the traits of that pop's species and extrapolating the makeup of that specific pop – in fact I believe Vicky does exactly this. And the most clear reason planets absolutely could not just be replaced with a "development" score is species traits. The whole point of the post was to remove discretely simulated pops without removing the features that come along with them in general and species traits in particular. They would still impact your production ("32% of your Clerks are Thrifty, giving you an extra 8% trade value") and your ethics. About 95% of the game would be the same as now (at least if you don't include "specialisation" as a modifier and just set it to 1). In the back-end it would be fairly similar to Vicky – Paradox' most in-depth politics sim (though everything turns fascist there) – and Democracy – the best known political sim currently. At the scale of an empire, what is an individual anyway?


Cristokos

>I don't think there are a whole lot of ethics-based pop-level modifiers, are there? Originally, there were. Materialists, for example, were better at generating science. This was abandoned because sector governors couldn't micro it properly and wouldn't have worked after Utopia anyways.


Darth__Vader_

Hey, as a CS student I really appreciate the in depth ideas here. I like the idea of getting rid of discrete pops, but I think we can do both. Say when you get 10 pops, they are then treated as a decapop (DP) instead of a normal pop (P). This decapop's residents do not check for new jobs as long as there is at least one normal pop doing the job. For example on a large world where there are 52 energy creators, that is 5 DP, and 2 P. The two pops still check for jobs as normal. However the DP only check that there is at least one P doing the job. This changes the number of checks on a normal world it would be O(n\*k\*j), whereas the decapop system is O([N%10]\*K*J) where here J is the number of normal pops employed. This means that the time complexity now depends on primarily the number of types of jobs, and that will rise more slowly. How I would imagine this algorithm working is 1. Normal algorithm of normal pops which will be on average (5*T) where T is the number of different jobs. 2. It then checks if any job has 0 P working on it, if so it breaks one DP in that job into 10 P. 3. The system then checks if the number of P working on any job is > 10. If so it condenses them into DP, leaving P%10 left. So the final time complexity will be (T^3 ). However T is dependent on N, and we know that job types increase with the population of the planet. I'd argue a good estimate for this is log(N), meaning a planet with 150 pops would have 15 different types of jobs. So the time complexity is O(Log(N)^3 ). Which is logarithmic time complexity instead of polynomial time. However this does come with some issues species being the biggest one I see, they can preform different amounts of work. This can be solved I think by taking the average production of the members of a DP when it if first formed, and then treating it like it's own pop with a modifier of that value. For example if species one does 1 unit of work, and species 2 does 3. Then a DP formed with 4 of species 1, and 6 of species 2, would be treated as a pop that does [4*1+6*2] or 16 work per month. This reduces the calculation needed, instead of 10 economy calculations there is only one, however this does present a problem, as on large maps where the population is in the tens of thousands, those values would start to take up space. One method to help with this is to bias the selection of DP to have only 1 species as often as possible. Perhaps once every 10 years the system rearranges the DP similar to de fragmenting a disk. To do this it breaks ones that are not marked as one species then reforms them. This would cause a lag spike for a single moment, but it would then reduce the number of unique DP and increase the number of uniform DP. The reason we need to care about the possible resource consumption of the DP is that there are a huge number of possible DP. Given 45 species as above there are a possible (45+9) C 10 combinations. Thats (45+9)!/(44! * 10!) or 23,930,713,170 then assuming they store the value as an unsigned 8 bit integer, the best case scenario this is 24 GB of ram needed to store the values alone. However if they want a more reasonable and accurate system and store it as a float that's 32 bits per value, meaning that the total ram needed could be up to 98 GB. This is also assuming that there is only one unique value, but the system would also need to keep track of how many of each species there were, so that number would grow fast. So we would need a method to prevent every combination from being created. My method would take the disarray produced in those 10 years, (the number could be made longer or shorter), and then makes them into a large number of uniform DP and a few very chaotic DP. The value for DP is far more compressed, an int for species type, and maybe a float for the number of DP. Since each of these DP is uniform, then they don't need to be differentiated in the job market, so we don't need to give them ID's or something. To keep track of pops, give each DP a unique identifier, and assign a pop that identifier when it joins or leaves a DP. This system reduced the number of unique DP to (U*S)/10 where U is the number of P that don't fit into a group of 10, and S is the number of discrete species. If we say S(i) represents the population a species with identifier I, which has a range of 1 to 45, we can then get a better idea of this value. ([i=0] ∑^S [S(i)%10] *S)/10 This results in a worst case scenario of 41 unique DP per planet, however under normal conditions we can assume [S(i)%10] is 5, not 9. Giving us an average of 23 if every planet has 45 species. However that is unlikely as well, it is more likely each planet would have perhaps 10 species. Giving us a realistic number of 5 unique DP per planet. So our resource usage now grows linearly along with the number of colonized planets. This number will likely not exceed 1000. Giving us a total max unique DP number of 41,000, which is 0.00171328% the number of the possible greatest without it. At 41,000 with that same float value that gave us 98GB we now have 164 KB, which is vastly less resource intensive. That is the MAXIMUM going off our more reasonable estimate of 5 instead of 41, we have 20 KB to store the economy values of all the unique DP. Double that to include the makeup of the DP, and add S to account for the rest of the population. So the resource usage should grow now at a logarithmic rate with population, a linear rate with planets, and an linear rate with species. This new system is O(log(N)^3 \*P*S) where N is the total population, P is the number of colonized planets, and S is the number of different species. To further optimize edge cases bias in the deconstruction of the DP as well. If we now take a theoretical game of Stellaris, total galaxy population of 50,000, 500 colonized worlds, and 45 species, and plug that into the original equation N^2 we get (50,000^2) or 2,500,000,000. With the new one we get (Log[50000]^3 \*500*45) or ~2,334,482. This reduces the load by 99.9067 % at that point.


Nighthunter007

Argh, I'm mad that I don't have time to go through this properly until tomorrow; that was a fascinating read and a good pitch. I will say, my O(n^(2)) estimate is probably way high, as it assumes very little optimisation (some that I alluded to, but also stuff like only recalculating once something changes). I would put it likely a lot closer to the lower bound, so my actual double-bounded guess is Θ(nk) per planet. Since k is quite small and relatively constant (at about 1-15), at a galaxy scale it would be more like O(np). In your example calculation that's 25,000,000.


Hectagonal-butt

and then make a private sector and and expand consumers good to be 15 different products and put in the austro-hungarian empire and


ST-Helios

... don't forget to introduce the sick man of X quadrant of the galaxy


ChineseHotAirBalloon

> and put in the austro-hungarian empire and Wait, that's an excellent idea. I never knew how much I wanted Franz Ferdinand in space


Frydendahl

Why is there no spy mission to assassinate another empire's leader?!


EMSEADESIXONEFOUR

I ran an austro-hungarian empire once, did syncretic evolution using the bat like mammal portrait for the habsburgs with default humans as the sub-species. I roll-played it as the habs having inbred so much as to be a functionally different species.


Akasha1885

I mean, if anyone I count on Paradox to invent the wheel anew. But it's insanely difficult, just like inventing something better than the car. The Pop system is so popular and widespread, nobody wants to spent resources on a System that might be unpopular with gamers. Civilization, the Endless games, Humankind, Stellaris every new game uses pops. Probably because they are more accessible to the layman.


Nighthunter007

That's why I tried to keep "pops" as a concept that's relatively simple. I go fairly in depth with the runtime analysis and stuff, but you could still have "pops" that exist in whole numbers and do "jobs" that exist in whole numbers. As much of the difference as possible exists behind the scenes. Games like Civ, while they have individual discrete pops, those are all identical. Happiness is shared for the whole city, bonuses are shared for the whole city, religion is kept track of centrally (rather than each pop having a certain chance to convert each turn). Outwardly it could feel very close to the pop system. Of course, you could also take this idea and run with it into full Vicky 2 pie chart mode pretty far removed from pops.


DiabloGoreOrRiot

To add to your point, I think Civ doesn't even have pops as agents, but only city tiles. Tiles can be worked by a pop. Pops is just a number assigned to a city.


SinisterTuba

I don't know anything about coding so I'm curious as to why every pop needs to calculate for every job on a planet. Why not have it so if there are four available jobs, four random pops make the check, and if there are no pops, there's no check at all.


Nighthunter007

In the unoptimised case (which is what I'm presenting) there are usually some corners that can very much be cut without losing any (or very little) functionality. Essentially you're trying to make sure that the pop doing the job is always the best pop for the job (or perhaps that it is the best job it could be doing). If you pick random pops to fill jobs, you can end up with your Weak pops working mining jobs while your Industrious pops work in research – wasting the traits. There's a *lot* of room to optimise my description of the current system, but it's right in a sort of "approximate" way. It is right in saying that as the number of pops goes up so does the number of calculations (which is what O(n) means).


DanielPBak

Software dev here, I have some disagreements with your findings. Your runtime per-planet analysis seems off (at least if PDX implemented this well), because you've ignored a lot of potential for preprocessing and calculating based on parameters that scale less quickly than pops. ​ Two identical pops on the same planet do not need to compare the same job. You could instead compare \*each sub-species on the planet\* for each job, which obviously going to be a substantially lower number than the number of pops. The fact that xeno-compatability causes performance problems indicates that PDX does this at least sometimes. ​ Furthermore, if two species have been compared for some job on some planet, you can cache that result, only re-calculating it when some parameter affecting those pops' job performance has changed. Though obviously this wouldn't effect the asymptotic growth it would practically turn the calculation into O(jobs) in many circumstances by reducing the number of effective species drastically. ​ If Species A is found to be better at job X than Species B, you don't need to calculate the effectiveness over and over. All of job X should be filled by Species A, with only excess going to Species B. Every time a new Species A is added, the result is cached so we know that Species B should be fired and A hired. ​ So this is obviously not a calculation whose best-case is O(pops). This is more like O(jobs \* species). Which should be much smaller, at least on larger planets. ​ The fact that reducing the # of pops improved performance a lot is NOT, to me, an indication that agent-based pops are a bad concept, but rather that more effort needs to be made to have the code scale off of the number of species and jobs rather than the number of pops on some planet. There tends to not be too many species, and many sub-species perform identically at many jobs and can be clustered. It's just a more complicated implementation.


Vaperius

> You could instead compare *each sub-species on the planet* for each job, which obviously going to be a substantially lower number than the number of pops. The fact that xeno-compatability causes performance problems indicates that PDX does this at least sometimes. I don't even need to even reallydo that, if the devs were willing to sacrifice a bit of the "magic" for Stellaris and make all pops being treated 100% equally as a resource of sorts you could make it so a "pop" becomes a "population", I've had an idea and I'd love for a software dev to give me an opinion. So hmmm System takes an average of the pops on the world and determines an approximate impact those species have on X production, basically rather than a single "pop" only having one job, every population has a base level contribution to planet wide production with their traits modifying their contribution in specific industries by determining their overall availability to the workforce based on their total population proportion. So if two species have a trait and are the only species on the planet, that planet has 100% accessibility to the bonuses associated with the trait, otherwise they receive only an average applied to the production on the planet. In this way, you don't need to check species more than twice, once to check if they are allowed to work that job type and once to tally up the trait total available on the planet. So that would essentially the system treats all species populations as the same *generally* for their "workforce availability" barring species rights differences based on their total population share. So rather than discrete agents, each population contributes to the available workforce in the abstract effectively, with as few pop specific calculations as possible. So at least as far as I conceptualize this, the only checks being made are check for how many of a species are on a planet(two checks?), their traits(average of five checks, once per species), how many copies of those traits are on the planet, the total population of those traits as an average on the planet, and the system then just fills "labor demand" across all industries based on a weight system that generally adds production in favor of higher stratum; so two checks per resource type produced compared against available workforce and planetary production modifiers(traits, generally). Compare this to the current system which individually calculates a lot of these checks already for *every single pop in the galaxy* rather than at a macroplanetary level, and its not hard to see why this game chugs by the 2000 pop mark, nevermind the 10k pop mark.


Nighthunter007

My estimations are certainly on the higher side. I kept the essentially "worst implementation that would work" estimate of O(n²) around, but it's unlikely. On a galaxy scale, something more like O(n) (or O(jobs), which should be similar) is a much better estimate, with some serious work to save computation time. This post is mostly an exercise in trying to decouple runtime and pop growth completely on a high level, which might not at all be necessary (though Paradox hasn't come up with any complete solution yet), but is nevertheless an interesting exercise. There is, after all, a practical limit to how fast it needs to be and how many pops it needs to support. Nevertheless, there's a ton of work gone into pops at this point, and the more we delve into how they might have optimised it the more speculation it becomes. I sort of threw out a range of estimates to cover my bases a bit; I'm much more confident in the estimates of my alternate design.


Aazadan

> The fact that reducing the # of pops improved performance a lot is NOT, to me, an indication that agent-based pops are a bad concept, but rather that more effort needs to be made to have the code scale off of the number of species and jobs rather than the number of pops on some planet. There tends to not be too many species, and many sub-species perform identically at many jobs and can be clustered. It's just a more complicated implementation. There are two factors here that modify this a bit. One is planet habitability, the other is planet modifiers. A tropical world, a Tundra world with high quality minerals but no Mineral Plant, and a desert world with poor minerals and a Mineral plant are all going to be quite different even for the same subspecies.


DanielPBak

the comparison is within a planet. This is a per-planet calculation obviously. Results would be cached per-planet.


Averath

How many years has it been since pops as agents were introduced? How many years has it been since we've had terrible performance as a consequence of this decision? How many attempts have they made to fix this problem, and utterly failed? I would argue that agent-based pops are a bad concept, considering how much they've invested into trying to alleviate the issues they cause, and how many times they've failed. While it is true that this could be due to their code being inefficient, that just illustrates that PDX is incapable of developing a system in which agent-based pops would function. If a soulless corporation keeps trying to build a toaster, but fails to make it actually toast bread within a couple minutes and it instead takes an hour, then they probably shouldn't be making toasters. And we, as the consumer, shouldn't defend their inability to make a toaster. That actively hurts competition by excusing areas in which they need to improve.


DanielPBak

You dont know what youre talking about


EroViceCream

Damn it, one hour to toast! I guess I should go take a bath and play a little stellaris, its an hour, but it's the beat toast you'll ever eat!


Averath

If that's the >beat toast you'll ever eat! then just wait until you have some toast from a functional toaster. It'll blow your mind.


EroViceCream

Where is it then? Talking is easy, but I see no results from your side.


Averath

Results? What are you talking about?


EroViceCream

I'm talking about the functional toaster. Where is it? I have seen no toaster that toasts as good as stellaris.


Xyales

Starsector is a pretty good toaster, but it isn't 4x and less complicated in its colony mechanics, but its good either way. Its just a completely different game made by someone with basically no budget but plays better than most AAA-Games. They even ask if you actually meant to buy 2 copies of the game lol. I do not like it when someone shittalks my 1500 hours of usage on that specific toaster though, the company made the toaster good enough and i had plenty of fun. It just needs a little rework, here and there. But its relatively hard to add a better heater, because there isn't enough space in the chassis. So they need a custom heater, modified to fit in the chassis of the toaster that is called Stellaris. It is possible to modify the toaster to have a GUI and you can also replace a lot of its features and parts, so someone might someday add a better heater that is even better than all the other toasters! :)


Averath

There are a *lot* of 4X games out there. It's a *genre,* after all. Masters of Orion 2 is the original "best 4X space game". Then there's Sword of the Stars. Endless Space. Distant Worlds: Universe. Galactic Civilization. StarDrive, Star Ruler, Star Dynasties, etc. Are any of these perfect? No, absolutely not. No one has ever claimed that a game has to be perfect. They all have their pros and cons. They all have areas they excel in, and areas in which they're lagging behind or outright broken. The difference with Stellaris is it's one of those rare titles where loyalty to the brand is paramount. They have supported the game for years and toss out at least one major expansion per year, often with smaller DLC popping up in-between. This, as mentioned, builds a growing investment in the game and growing loyalty to the brand itself. After all, you've likely spent close to $200 USD on Stellaris, and if anyone starts pointing out the flaws of that game, they're threatening to suggest that you've wasted some of your money! And that makes people very, very angry! Stellaris isn't the worst game on the market, but it is not the Second Coming of Christ in video game form that some people seem to act like. It is a very flawed product that continues to be flawed, likely due to mismanagement from the top. Remember that people vehemently defended Cyberpunk 2077 and ***sent death threats to someone who pointed out parts of the game could cause seizures.*** They also defended Anthem, which is now dead. Stellaris can be a fun game. I occasionally enjoy playing it. But I also acknowledge that it is riddled with so many problems that have gone unaddressed for literal years. That applies to most games. I love the Total War series, and *those* games have a massive amount of problems. I can still play it for hundreds of hours. I love Monster Hunter World, and that game has *tons* of issues, but I still enjoy playing it. But I will never, ever pretend like games I enjoy are perfect or without flaws, or will provide me with the best experience. Because that would simply be me lying to myself to try to justify my purchase and not feel as if I was taken advantage of a corporation.


RaederX

I am not a fan of discrete pops and believe the population should be managed as a total with a percentage allocation across the jobs available and a way to direct the movement of percentages of the pop. That being said, if only one random pop in each job was to check to see if there was a better job available for it, and if there was (and presumably it the pop moved) a second random pop in that job could then also make that check and move, the the total number of checks and movements would be a small fraction of the 2.8.1 pop transition and this would solve the performance issue.


Space-Robot

This is impressively thorough, but I could tell almost immediately it was coming from a student or academic. The primary issue with the idea (and I recognize this may be irrelevant to the reasons you have for making this post) is that even if it were a given that it was the right change to do, would massively help performance, would make the players happy, etc, then it still couldn't be made reality because of the size of the change. * The Stellaris code base is no doubt huge. It was probably huge when the game first came out, and it has undergone 5 years of VERY active development since then. Even with expert developers, modern methodologies, and understanding leadership, the source code probably resembles a bunch of rubberband balls welded together with acid. * The population system is likely a core system, implemented early on and built on top from the beginning. It probably has its fingers in more places than any one developer at Paradox knows. It probably interacts with code that everyone on the team is afraid to touch. * Even if the last bullet is as untrue as possible, replacing the system entirely would still be SO, SO much more work than modifying the existing one. It's easiest to add stuff. A bit harder to modify stuff. A lot harder to replace stuff. I'm tempted to go into more detail about that but I could go on for way too long. Basically I would bet any Senior+ Dev working on Stellaris would wince at the proposition of replacing the agent-based system. If there's any room for optimization of the existing processes they will definitely explore that first as it could probably solve the problem "good enough" for 10% of the effort. * It wouldn't be worth it from a business perspective. Paradox is admirably good about releasing free content, updates, reworks, etc but they're still a business. They want to keep the game healthy and the fans loyal so they can sell more DLC and other titles. They can't really sell the fix, and they have to consider that Stellaris is already 5 years old and probably has more DLC behind it than in front. A "good enough" solution will beat a "best" solution in this kind of scenario. * This point is entirely subjective and much less relevant than any of the others... but agents are just more fun to work on than straight math :)


Nighthunter007

Oh you're absolutely right it was written from an academic, high level point of view, basically deliberately ignoring the actual code base. If anything, this is the kind of change you could probably only get away with in Stellaris 2 after burning the codebase to the ground. If they had the ability to implement this in a timely and affordable fashion I would have to call the Inquisition because that is sone serious black magic. Your points are all valid, and as you say the post kind of ignored this. I should probably have mentioned somewhere that this wholesale replacement of a core system is unrealistic, but this whole post is unrealistic at its very core. I mean, "someone suggested sweeping technical and gameplay changes to our game on Reddit? Better get to work folks."


wutzibu

I didn't understood the math but that's maybe because I am still zonked out by general anasthesia Still I like the idea to implement a vicky 2 like system. (but please ditch the market. The market sucks)


Nighthunter007

Hah don't worry about the maths; it's mostly just to be able to talk about "faster" without ambiguity about what I mean. Basically: the current system does a bunch of stuff for every pop, so when there's lots of pops it has to do lots of stuff. More pops -> slower game. My proposal does stuff per species, so even if the species has 10k pops the game is not doing any more work then when it had 10 pops. More pops -> no difference.


balne

I...I actually understood what you meant due to my college education. I'm so happy with myself.


Nighthunter007

Haha, finally it pays off!


TheGokki

Starsector does it the best i think. Population is treated as factor, and counted like "thousands", "tens of thousands", "hundreds of thousands", etc... All that matters is how many zeroes the overall population is, there's no difference between 12500 or 99500, only wether or not it has five or six digits. Population could, this way, be counted in the following "tiers": * 1,000 (starting non-functioning colony) * 10,000 (starter colony that finished colonization procedure) * 100,000 * 1,000,000 * 10,000,000 * 100,000,000 * 1,000,000,000 (real life Earth is here, soon approaching the next tier) * 10,000,000,000 * 100,000,000,000 (Ecumenopolis can now begin construction) * So on and so on... So in the above example there's only 9 population levels, pop growth simply takes longer between each level but growth should remain the same (as devs want to balance it). Gameplay shouldn't change much with this, it's just a numbers thing.


Xyales

Right, i've been playing starsector a lot recently too and its true that it has a well made colony system, though it isn't as complicated. One could "fix" Stellaris Colony System by simply removing a few features like Unemployment and Job Capacity and just use the Max Industry and Unlimited Jobs and Statistic Species Calculation instead of per Pop basis. There have been posts that undermine the intensity of Starsector colonies, but they don't take into account that a size 9 colony exports billions of tons of goods per month compared to a size 4 colony, even though they have the same industry building. The Colonies Infrastructure and Industry scales with the colony and Stellaris fails to implement this by having a lot more slots and just repetitive spamming of buildings and districts. They should remove District Caps and make all Tiles and Blockers affect production instead, everything that gives Industry size should be instead correlated to Mineral Purity or Production Possibilities. It is obvious that it will take an eternity until Paradox implements such mechanics because its a big change, so instead it would maybe be better to implement what OP described or the Starsector variant as a mod. ... This Post makes me actually want to create such a mod, but sadly i am busy with exams :(


Ferrus_Animus

IIRC it is less the pops that look for new jobs, but jobs that look for pops to fill them. Which is why overbuilding increases lag. I ahve wondered if a simply solution wouldn't be to only have one job per planet check each day and slowly advance through them. This would cap job calculation to number of inhabited planets, reducing one of the variables to a simple 1. The downside would be that bigger planets would need multiple months to reorganize pops, which to me seems acceptable (and wouldn't break versimilitude).


Diogenes_of_Sparta

This very thing was proposed and rejected when 2 2 happened and people threw a shitfit over the change from tiles.


Nighthunter007

I can imagine. There's always someone throwing a shitfit somewhere, isn't there.


Diogenes_of_Sparta

[Yes.](https://old.reddit.com/r/Stellaris/comments/8wa30p/a_teaser_from_martin/e1tutyl/?context=3)


Little_Chick_Pea

Actually I'm pretty sure that it's jobs that's look for pops, not pops that look for jobs. I can't remember where I read that though.


etra_kurdaj

What if instead of EVERY pop being mobile and movable, only 5-15% of pops were mobile and movable? This represents entrenched communities that will never leave and die defending their homes. The most mobile populations are young people and early families. So you have a population of 85 on a planet, but really only 9 or so of them are in a position to consider new opportunities. The remaining 76 are left out of the equation because they're "entrenched".


Cabinet_Jaded

I filled an ecumenopolis. Well 3. It can be done.


SkyF1r3-90

Great work. Here’s to hoping PDX puts out a quick hot fix to let our pops bone a bit more!


ReMeDyIII

I found a [mod](https://steamcommunity.com/sharedfiles/filedetails/?id=2458663897) that changes Paradox's defines.txt file. I know this isn't the best solution, but if we're going to make the best out of a bad situation, in your opinion, what is the perfect amount on these three defines for a Large galaxy size?: **MAX\_CARRYING\_CAPACITY = ?** \^ Default = 500. Max pop. **REQUIRED\_POP\_GROWTH\_SCALE = ?** \^ Default = 0.5. Required pop growth increased by this much for every pop in the empire. **REQUIRED\_POP\_ASSEMBLY\_SCALE = ?** \^ Default = 0.5. Required pop assembly increased by this much for every pop in the empire.


Nighthunter007

Honestly, idk. Before it crippled performance. I've seen (and tried) mods that set the scale factors to 0, but that just brings back that same late-game crawl.


Astornautti

You guys are getting a pop every 3 years? My planets take 2 decades to grow a single pop and I have barely reached endgame.


HrabiaVulpes

Okay, I mostly skimmed through it since holy shit full on kudos to you for writing that much. I thought about this many times before, so I will just share what problems came to my mind over time: 1. **Ethics-per-pop** \- currently when you conquer people of different ethics it's a mechanic that they are a problem at the beginning. They are creating factions that differ from your government ethics, are unhappy about many issues etc. Solving it by "ethics-per-species" is not a solution because it completely ignores that we can have more than one empire with the same species and differing ethics (and that ethics are not something you are born with). 2. **Modifier-per-pop** \- there aren't many of them, sure, but they still exist. Not a big deal since they will come up at most once per few games. 3. **Shifting the problem** \- Previous per-pop calculations would now be made on species-per-stratum. There are strata modifiers so we can't remove those and there are inter-species differences. This would make every specie has effectively 3 pops on each planet. Xeno-compatibility and biological ascension are still bane of performance. 4. **Partially filled job**s - Lets say we treat 1 billion people as we treated one pop previously. We have three jobs, and two species 1.5 billion each. Which specie gets third job? My personal solution was that jobs are no longer discrete and planet that has for example: 2 colonist, 2 entertainers, 1 metallurgist, 3 technicians, 3 farmers (10 jobs in total) simply employs 20% of people as colonists, 20% of people as entertainers, 10% of people as metallurgists etc. Of course that changes very core systems of the game that makes playing tall more potent. 5. **Population growth** \- currently each time pop is finished game chooses which pop should be produced next. With constant stream of non-discrete population there would either be growth of all species simultaneously (making multi-specie empires vastly more powerful) or one specie could grow forever by growing and emigrating to other planets at a constant rate. 6. **Pop-based modifiers** \- There are some modifiers and triggers that come per-pop. When pops are discrete it's blatantly clear how this works - upgrade a building when 10 pops are on planet, but what about when you have 9.99 pops on planet due to numerical growth? In general - removing discrete pops and turning to more fluid approach with numbers would probably be Stellaris 4.0 and honestly it would change the game even more than any previous big update. I don't believe it will ever happen (it would throw out a low of work they already invested in the system twice), but if someone would like to make Stellaris-like game with this I would certainly try it. Too big of a project for me personally... Systems that would have to be changed: * How pops grow * How pops migrate * How jobs pick pops * How ethics change * All pop related events * All modifiers coming from amount of pops doing X


Dr_Zorand

> So this means if 10% of your pops are Industrious, you get a total of +1.5% mineral output from jobs. At 100 specialisation, we assume every species works at something they are good at if possible. Thus, your mineral output boost is determined by the number of Industrious pops you have divided by the number of miner jobs you have. So for 15 Industrious pops and 50 miner jobs your total would be +4.5%. A potential problem with his approach is how you handle pops with multiple traits. Imagine you have a planet that is 1/3 human and 2/3 blorg. The humans are both industrious and agrarian, and the blorg have no useful traits. If the plant is 1/3 mines, 1/3 farms, and 1/3 energy districts, your system as you describe it would look at it and say, "1/3 of the pops are industrious, and 1/3 of the jobs are mining jobs, so full bonus there. 1/3 of the pops are agrarian, and 1/3 of the jobs are farming, so full bonus there." But that would mean that 1/3 of your pops are doing 2/3 of your jobs because they're being double counted.


Nighthunter007

That's a good spot! I talked about it in another subthread after realising it: https://old.reddit.com/r/Stellaris/comments/mu4mvr/an_indepth_discussion_of_the_pop_system_its/gv4zv7d/ You *can* fix that leak without losing the O(s+k) runtime, or you can actually assign the jobs to species types, get some extra perks like less cheating portraits, and only drop to a O(s\*k) runtime which is still pop-independent and might even be faster in practice (you'd usually go from checking about 5 jobs with matching traits items per species to about 10-15 of all jobs, so not a huge deal).


[deleted]

Unpopular opinion: remove all formulas, let me drag and drop pops. Boom.


DiamondNova

Ye, good luck with thousands of pops in late game


[deleted]

How would that be hard? How often do you really see the need to move pops after initially placing them?


Aazadan

I've thought about a lot of this before (game dev here), and there's a lot of ways where this sort of system could be emulated, keeping about 95% of the functionality without needing discrete pops. Pop count has been an issue for a while so I've had a bit of time to think about it. The pop number shouldn't be an issue though unless it gets too high. I'm skeptical it's running in n^2 though, because with some good sorting you can get it down significantly. Probably to N * log n. And neither of us has seen the source code, but for the most part pops shouldn't need to be checked each check. It should be possible to simply use a series of cascading flags. If a pop is unemployed, notify the planet it's on, then the planet notifies the empire and includes a reference to the pop. Then, if the empire has a flag tripped at it's check, it goes to the planet, determines if it can make a change, and if not, then goes to the pop to move it. Basically, a set of state machines. I'm busy at work at the moment so I can't go into more depth on this. I suspect they're not using this sort of system, but I think they're also using something considerably more efficient than what you're suggesting. The reason I think this, is that simple observation can confirm it's not running in exponential time. You can look at the computation speed changing per cycle and for example a 30k pop galaxy isn't slowing down at an exponential rate from a 15k pop. That said, pops are just a number and the current system is pretty good in my opinion. People just have to get used to change and figure out how to modify strategies.


Nighthunter007

I agree it's *probably* not running in O(n^2) time; that would be the naïve implementation (and would have hopefully been optimised away early). It's much more likely to be O(nk) imo, but I put both my limits up there anyway. Actually my estimates were (perhaps not clearly?) per planet, so galaxy-wide it'd be O(n^(2)/p) (which is still quadratic and probably too slow, also p is only like a few hundred at most so that's basically O(n^(2))) or the more likely O(n/p\*k\*p)=O(nk), where k is now so small and constant that it essentially becomes O(n). I'm also sure they've optimised the hell out of the pops system. I remember some talk about it in fact in a dev diary. I sort of presented the unoptimised procedure. Regardless, the runtime clearly grows with pop count. You can do a lot of things, and I'm sure they have, to optimise the real-world performance like not recalculating unchanged planets or pops (there is some evidence of this in how you can close and reopen jobs and the pop distribution after is different than before despite the conditions now being the same, which suggests the system reacts to changes rather than ever calculating from scratch). None of these remove the underlying issue that as the population grows the galaxy slows down. They would also work out (I think) to constants, so it's still O(n). I skipped over all of this simply because it isn't working well enough. They've presumably done great work on optimising it, but it's just clearly not enough without also Thanosing the galaxy. Add to that that it would all be (even more) conjecture and speculation and I figured I'd leave it to discuss in comments ;)


Aazadan

They’ve put a bunch of optimization in. The real problem is that they’re committed to making each pop a distinct entity. On the one hand it’s cool but on the other hand it’s very inefficient. If it were me and I was on a team that really, really wanted the individual unit portraits, I would probably assign each job to hidden values for these, and show them when the job is full. Then handle everything behind the scenes in a more efficient manner. While probably leaving leaders as distinct units. And basically fake it. Edit: I’ve messed with the specialization idea a bit too. As distinct units you can still do this really efficiently. Basically, you need traits to add points for job preference. Then, keep track of a pops current job. Then you check pops periodically for unemployment and under employment. One is no job, the other is a job it’s less suited for. From here you use the preference from perks (like mining output adds mining preference) and if one is available you find it and add them. This should require an empire wide check for each planet to sum all open jobs. And a scan of all pops for employment status. Then you can just do a naive match really quickly. Not 100% optimized, but good enough while leaving another 10 to 15 percent in gains available for those who like to micro manage.


Takfloyd

No, no and no. Pops as discrete "persons" is at the heart of Stellaris. It's what gives the planets flavour. Turning them into a number on a spreadsheet is the worst thing the game could do. And pushing them in that direction is why it was wrong to move away from the old tile system with pops on them, too, since now the pops aren't visible on the main planet interface anymore.


Nighthunter007

I wholeheartedly disagree about the tile system, but it's a fair opinion to have. In some other comments we've discussed how the system can outwardly appear very similar to today's discrete pops system, but it would still be a step even further away from tiles.


IndustrialistCrab

Can someone make this into a mod pls?


Nighthunter007

I don't think that's possible, sadly. This would go way beyond what is exposed in the modding API. I guess it might be *possible*, but only by reverse-engineering the source code (in which case let's hope PDS didn't take Enigmatic Engineering).


IndustrialistCrab

Should I feel guilty that I always take Enigmatic Engineering?


Nighthunter007

You've doomed us all.


IndustrialistCrab

Well, as a Determined Exterminator, I'm proud of my good and efficient work towards total doom.


EXFO-1945

I feel that eingmatic engineering shoul be circumvented by brute force research. No amount of encryption will stop a determined scientist.


UristTheDwarf

Perhaps, but the scientist that tries will probably go mad in the process.


EXFO-1945

Im pretty sure he was mad to begin with. You kinda have to be to even try to understand enigmatic engineering.


ReMeDyIII

My concern is the devs are clearly aware of the issue, but rather than fix it, they took a shotgun approach and were like, "We're limiting pops!" If the devs do not have the expertise to fix the issue, then we're wasting our time unfortunately. A modder would have to fix it, unless it's hard coded. Edit: Also, I'm a fairly new player still, but could anyone with an ~5ghz CPU play Stellaris with no slowdown before the pop changes? I'm just curious if excellent hardware could overcome the inefficient pop system pre-dating Nemesis.


Aazadan

Depends on how large the galaxy got. I could get my games to taking a minute to tick a single day at the fastest setting without much difficulty. Just by using certain builds it was basically a given to happen.


residentmouse

It's really cool to see how passionate you are about problem solving. I think it speaks well to your technical skills, and applying them in this fashion is a great exercise! But even though you acknowledge this isn't your domain, I don't think you *fully* appreciate how far out of your domain this is, or how extremely unlikely it is for a Paradox developer to read this post and gleam *any* insight*.* An *extremely* important skill to realize is that a discovery for you, a **brilliant** idea for you, can be a 1-second intuition for someone with experience. This isn't just true for tech, but it is **especially** true for tech. I don't want to be harsh, but there is nothing in this post that is novel to anyone working in the codebase. "What if we abstracted population in order to improve late game performance?" **could** spark a conversation, but that conversation moves past the details of this post into trickier problems and game design concerns almost immediately. I say all this not to be critical, but because the most important lesson to learn is this: If you are the smartest person in the room, you're in a room with nothing to offer. And you're sure as shit not going to get into the room you *want* to be in without a little empathy and humility.


Nighthunter007

If you read this as lacking empathy and humility then I have misrepresented my intentions. This is not addressed to Paradox (though it would be interesting to discuss this topic with them, into the further topics of taking a fairly vague idea with some thought into an actual discussion of the tech and game design), but mostly to share into the room some thought that I spent some time writing down. I specifically approached this from a tech perspective (as that is my primary domain), and I fully acknowledge that game design is no more than something I yell at the TV (well, PC) over (that's partly why I kept the pops as close as I could to the current game – so I could do as little game design as possible). I also know full well that I spent a few hours looking at a thing people probably smarter than me work on for a living. If I came across as a know-it-all or as "they should just do this simple thing I can't believe they've thought of" I sincerely apologise (though more, I suppose, to the hypothetical Paradoxian reading than to you).


residentmouse

All good mate, honestly I don’t think anyone was offended. The crux of the issue is you’re biased towards a gameplay decision, and you’re making that a *technical* problem while (despite what you say) absolutely addressing Paradox employees - several times! It’s a small part of your amazing post but it’s still a bizarre thing to witness. In your reply you *again* mention being interested in talking to them! At this point I have to give up. The empathy comes from imagining yourself as a dev in this theoretical situation where they reach out to a comp-sci student who is unhappy with their latest patch, and has written a 30,000 word essay analysing a *theoretical* function’s replacement based on interpreting Dev Diary’s like some Rosetta Stone. You keep saying you want this scenario to happen so *imagine* what that looks like. What on earth does that person say to you other than “I’m happy you’re passionate about our game and I’m sure you’ll get good grades on your intro to computational algorithms”?


Nighthunter007

I mean, the fact that it would have been cool to talk to Paradox about their tech and game design is, I think, something most people who read the dev diaries would agree with. That's why we read the dev diaries. For *me* it would be cool, because it would likely be a very interesting discussion (or, like, half-lecture with questions). That was what I meant by my parenthetical. I am under no illusions that this would be a particularly interesting discussion *for Paradox*, unless some of them just enjoy explaining stuff for it's own sake (and somehow decided to indulge a random person rather than writing a dev diary or something). I'm basically saying if I won an office tour in some giveaway or something I would have gone. The address to Paradox was kind of just a formalism. Once I've written 3300 words and shared it with other hopefully interested fans in a place Paradox can, at least in theory, read it I might as well add a sentence about that fact.


Averath

While you're right on so many levels, there's one major factor you've not mentioned. Leadership arrogance. It isn't just a matter of something being a brilliant idea for you, and a 1-second intuition for someone else. A prime example of this is basically the entire video game industry over the last few decades. But allow me to offer a prime example: BioWare ***Magic***! Or a situation where the leadership completely fails their teams because of their unrivaled arrogance. In the case of PDX, actual coders have written Dev Diaries before and stated how they could fix the problem, but they don't want to step on the toes of the devs. This should be a massive red flag to every gamer, but it flew under the radar and people just ignored it. The fact that we had a recent Art Dev Diary where the Lead Artist's designs were chosen over all of the other designs should be another massive red flag. When the Leaders of a team decide that they're the smartest person in the room and no one else has anything worthwhile to tell them, you have issues where teams start to breakdown. Unfortunately, the Gaming Industry has a way around this. They invest a massive amount of money into marketing to compensate for this. One needs to look no further than BioWare and CD Projeck Red. Anthem was a disaster, but they built up so much hype that their fans were willing to Stan for a soulless corporation that doesn't care about them. They were willing to go at each other like piranhas. Cyberpunk 2077 was not the product CDPR advertised, but when one journalist stated that the game had the potential for causing seizures, she was sent ***death threats***. The main reason that Leadership in the Gaming Industry has no checks or balances is because we, the consumer, enable them. We defend them. We give them excuses. We offer them patience. And they simply don't deserve it. All throughout Stellaris' lifespan we've seen signals that the Leadership is slowly destroying this game with their arrogance and unwillingness to accept that an experiment night not have worked out. They prioritize their own opinions over the work of their own employees. We have all of these negative signals that are warning us of just how bad things are, but we ignore them and anytime anyone steps out of line, the Stans rush in and silence any dissent against the soulless corporation they stand for. I don't understand why, though.


drbooker

You say that they're slowly destroying the game, but I've been playing Stellaris since its initial release, and have, embarrassingly, more than 2000 hours in the game. So far, every major update that I can remember has improved the game, and I think the fact that I'm still having fun with this game 5 years later, despite getting bored of it on several occasions, speaks to that. But I guess I'm just a Stan for a soulless corporation.


Averath

You didn't say "ThEn gO pLaY AnOtHeR GaMe!" so, no. I wouldn't necessarily say you're just a Stan for a soulless corporation. If something exists, you can go out there and find someone that likes it. Even if you feel it is the absolute worst thing on the planet, there is someone out there who will enjoy it. Stellaris isn't without it's merits. There are some good aspects to Stellaris, which is why I'm still on this subreddit. If Stellaris was 100% a piece of garbage, I would have given up on it a long time ago. It's because of the many ways in which Stellaris can be fun that I am so frustrated in how many other ways it fails to be fun.


kittenTakeover

Yeah, they should have done this from the start, but they got attached to the romanticism of tiles and individuals with "personality." They've been building on a poor foundation since.


EXFO-1945

I think you should try and change the pop to person count from 1 billion to 100 million, since im pretty sure we wont be having 23 billion people living on Terra in the next 179 years.


UristTheDwarf

That's what people said about there being 7 billion 200 years ago.


Nighthunter007

Well then we'd have 70 pops right now, which requires a bit of rebalancing of gameplay. You know, "it's 2200 and there are only 2.3 billion people on earth? What happened!" Easier to but actually worry about the number I think and just call it pops.


EXFO-1945

Well, s to what happened on Earth to have such a small population, the story does tell that conflicts and wars have happend, possibly even WW3, so thats not ganna contribute much to our population growth, not to mention birth rates in first world nations have dropped to the point of their populations declining, and this trend might happen across the world as more countries develop to the same level as first world nations. All this would mean that Earth's population being at 2.3 billion isnt all that far fetched. At least to me.


POKEMONMAN1123456789

Victoria 2 in a nutshell


PachoTidder

My monke brain can't handle that kind information


Katnip1502

looks like someone needs a good uplifting


PachoTidder

Yes please


Katnip1502

just step into this ship on the way to Pamper Wo- I mean the uplifting center!


Polari0

This is pretty much the kind of idea i have been thinking of since economy rework.


FourteenFour

Originally I was of the idea that we assign pops between worker and specialist jobs using a slider which determines percentage. This percentage is then applied to the total worker and specialist jobs available and output is created on that. So if you have 10 specialist jobs worked by five pops you get 50% of all specialist possible output - you don't choose which one is filled. You can tweak this by disabling a building or district on main planet screen; not individual job level. There is no demotion time in this system; something I think needs to be done away with anyway considering there are still bugs which can give you too many specialists and you get stuck waiting for them to demote. Then from that output you can apply specie bonuses using a similar method. This really does drop specie specialization on multi specie planets a lot. Meaning that unless you force a specie into one role it could take equal number of jobs of specialist. Hey, we have slave traits but we could also on the specie screen weight them to worker or specialist jobs. The second idea was to change when the game even looks at pops. Pops management per planet should only execute when a pop is added or removed, when specie is changed, when a building/district is added, changed, or removed. At no other time do we care. Once in a position it should be locked until the situation changes. To make the system smooth again I remove promotion and demotion times. Right now it looks like the code executes job assignment monthly even if nothing needs to change but it certainly will; I have seen all my drone/clerk jobs fill and deplete magically - part of the code decides they should work somewhere else one month and then suddenly it recognizes that oops - we needed them there in the first place. The third idea is similar to the first idea except we have more categories. We break jobs down by more categories. * Manufacturing jobs are those which transform one physical resource to another * Specialist jobs become those which provide non physical resources, think research points, unity, etc. * Gathering jobs are your miners, farmers, and technicians * Service jobs are your amenities, soldiers, security, and such. We then apply the slider system to put a percentage of the population into each category. Then from that output we apply the specie positive and negative traits based on percentage of planet population. there are no individual jobs, if you don't want a building or district to produce you disable it. All I know is the current solution to pop growth makes the game feel sluggish and growth unnatural and honestly that ruined the game for me


Usinaru

Amazing work. I wish this was implemented.