Non-Uniform Distribution of VS Weapon Traits
Authored by @kneewoah, Bungie: Newo#9010
Last Edited: Wed Oct 23 at 1:30pm EST
Fair warning, this document was partially written with genAI so I could write it faster. I’m a busy student, sorry. All the ideas and conclusions are my own.
Foreword
This report describes how I discovered a potential bug in Destiny’s loot system using a crowd-sourcing loot drops tool I developed. After noticing strange patterns in perk drops, I decided to collect data from players using a website I built to track new drops directly from their inventories. By gathering clean, reliable data, I was able to test whether the game’s perk distribution system is functioning as intended.
Motive
It all started last season with my friend Boop, who spent an absurd amount of time grinding for a serviceable Multimach CCX roll in Iron Banner. He opened around 1000 engrams [1], yet the roll he wanted never dropped, even though the odds were supposedly 1/36. This is where the “perk weighting” conspiracy gained traction.
When the season launched, many players were successful farming for a different weapon, Bitter/Sweet. This Grenade launcher had 7 perks in each column, yet anecdotally, it took most players a relatively normal time frame (<3 days, prior to the dungeon launch) to acquire their 1/49 rolls.
After Vesper's Host dropped, things changed. The Multimach CCX pattern repeated itself with the new dungeon grenade launcher, VS Chill Inhibitor. The grind for this one feels nearly impossible. Despite thousands of encounter clears, even with a presumed ⅙ drop chance on the gun (3 armor pieces, 3 weapons), and a 1/36 chance of getting the ideal perk roll, a rumor spread that no one was getting the roll—at all. It seemed like more than just bad luck. So the community took action.
What Went Wrong in Previous Attempts?
Last weekend, Spark, another player attempted to investigate this issue. He collected data from 13 players who hadn’t yet gotten the grenade launcher with the perks “Envious Arsenal” and “Bait and Switch” [2]. Unfortunately, this data collection method is inherently biased towards the null hypothesis. The data was skewed towards players who did not have the gun. The community deemed his findings inconclusive, but the idea still remained out there. This made it clear we needed a better approach to understand what was really happening.
My Hypothesis
My hypothesis was as follows: the perk distribution on the VS Chill Inhibitor is non-uniform. I believed the odds of getting certain combinations of perks were likely much lower than the game led us to believe, which would explain why no one was getting the rolls we wanted, even after hundreds of attempts. The only way to test this fairly was by using more controlled data.
My Methods
Instead of relying on self-reported data, I built a website that tracks new drops directly from a player’s inventory. The website design specifically required players to opt in to a session, meaning they were aware of what we were tracking, and I made sure to notify them not to delete any rolls before the data was collected.
Once the tool was activated, it would log any new items entering a player’s inventory during the session. This gave me a clean slate of data from the moment the player opted in. Each individual drop was assumed to be entirely independent of the other. All items, keyed by their official item instance id, were stored in a database for analysis later on.
I chose not to measure the overall drop rate. I simply wanted to investigate the distribution of perks, and overall gun drop rates are irrelevant (as far as we know) to this.
For the majority of this analysis, I am only focusing on the 3rd and 4th column traits (“perks”) of the weapons.
Figure 1
Screenshot of the GUI for players who are in a tracking session
Website: https://d2-loot-tracker.vercel.app/
Full source code: https://github.com/owens1127/d2-loot-tracker
- Recommended viewing: Scraping Logic, API Routes/Queries
Why These Methods Remove Bias
By using an automated tool that tracked drops directly from the player’s inventory, I mostly removed the human error and bias that comes from self-reporting. This ensured that I was not accidentally omitting or mislabeling rolls. Players also had to opt in and be aware of the tracking process, which made them less likely to accidentally delete valuable data.
Another strength of this method was that the tool only recorded drops after it was activated, so we had full control over when and how the data was being gathered. This eliminated any chance of including old drops that weren’t relevant to the current session.
Each weapon drop is an independent event. I am simply logging the results over hundreds of these independent events.
Potential Biases
There’s still a chance that some players may have deleted rolls despite the warnings, which could introduce some bias. However, since players were notified ahead of time not to delete anything, I believe this was minimal, random enough, and it shouldn’t have a significant impact on the results. Also, players who don’t fully participate in a session might inadvertently skew the data, but the opt-in system helps mitigate that risk.
There are also a few guns which the player must have enhanced prior to the data collection, which effectively removes them from the loot pool. This is less than 1% of the data, so I do not believe it will have an impact on the results. I can confirm however, that none of the enhanced weapons were the elusive Envious Arsenal + Bait and Switch combination.
It is also of note that the software was built in a hasty manner in order to begin data collection as quickly as possible. There were occasional crashes and glitches. There are a couple reports of previously acquired weapons which should not have been trackable being uploaded to the dataset. I do not believe this will significantly impact the results
Results
The data we gathered using my tool revealed something interesting: certain perk combinations on the dungeon grenade launcher were dropping at rates far lower than they should have. After running a chi-square test on the data, we found statistically significant discrepancies in how often specific perks were showing up together. Screenshots from the tracking tool clearly show “dead spots” in the perk combinations, where certain desirable rolls are virtually non-existent, despite the odds suggesting they should appear more often.
The following data will focus on the two different weapons tracked, the VS Chill Inhibitor and the VS Velocity Baton. Most players were focused on obtaining the Envious Arsenal + Bait and Switch roll on the VS Chill Inhibitor
Note: All screenshots were taken from my tool at 10:01am EST, Wed Oct 23rd.
Figure 2
This chart shows the perk distribution (n=452) of all rolls acquired on VS Chill Inhibitor by the scraper. A gradient applies a color based on the expected outcome, where black represents below expected and white represents above expected–neutral gray remains in the middle.
Figure 3
This chart shows the perk distribution (n=409) of all rolls acquired on VS Velocity Baton by the scraper. A gradient applies a color based on the expected outcome, where black represents below expected and white represents above expected–neutral gray remains in the middle.
Figure 4
This figure represents the result of conducting a chi square test on the VS Chill Inhibitor with alpha=0.001
Figure 5
This figure represents the result of conducting a chi square test on the VS Velocity Baton with alpha=0.001
These diagrams provide strong evidence that the perk distribution is not uniform. The likelihood of this occurring by random chance for both guns is exceedingly low, well below any relevant alpha level. Since the expected value for each perk combination exceeds 11, using the chi-square test is justified.
The initial idea to graph the perk combination in a 6x6 grid came from Vendetta and Wauby. They proposed the possibility that the distribution of perks might be correlated to the cartesian distance between the indices in the Destiny Socket definitions, calling it “Perk Proximity Theory” (PPT). While I have not run any statistical tests to show if this correlation shows true, the gradient on the graph certainly suggests this might be true. My hypothesis still remains that the chance of getting any random roll is non-uniform.
I did explore the idea of PPT a bit further, specifically going full circle to Multimach CCX and Bitter/Sweet research backed this up when we compared the findings to Boop’s hunt for the Multimach in Iron Banner. To do so, I explored data from light.gg. Before I discuss this, it is important to note that the light.gg data comes with a bias—it only tracks currently existing rolls. Players typically only keep rolls they consider decent or worth keeping, so the dataset skews toward better combinations, leaving out weaker rolls that are dismantled without being tracked. Also note that light.gg only shows the top 8 most used traits, the other combinations were found directly embedded in the HTML of the site.
Figure 6
This chart shows the number of tracked perk combinations on light.gg for the Bitter/Sweet grenade launcher [3]
Figure 7
This chart shows the number of tracked perk combinations on light.gg for the Bitter/Sweet grenade launcher [4]
The data here doesn’t stand out as much to me. The Multimach CCX data in Figure 7 in particular does not show the expected rarity of the Kinetic Tremors + Attrition orbs combo as I anticipated. This could be chalked up potentially to the fact that Multimach has existed far longer than any of these other guns, and players have grinded for their specific rolls more, further skewing the data. There are some specific combos worth noting that are considered to be strong rolls which players electively farm for, here are the two most desired: Kinetic Tremors + Attrition Orbs, Target Lock + Dynamic Sway Reduction.
However, the Bitter/Sweet data in Figure 6 does appear to show the same diagonal gradient as Figure 2 and Figure 3, potentially demonstrating a similar cause for the non-uniform distribution. I will not be drawing any conclusions from this light.gg data, but it is an alternative approach worth mentioning.
Further Actions Needed
This discovery has raised several important questions:
- Is this skewed perk distribution limited to the dungeon VS weapons, or is it a broader, systemic problem across other weapons in the game?
- Has this imbalance been present in the RNG system for a while, possibly affecting drops from other activities and loot sources?
- How would these conclusions vary on guns with more than 6 or 7 perks, should ritual playlist weapons which have 12 possible perk combinations and can roll multiple at a time?
- What is the actual cause of this problem, is there a correlation to be found between the data collected and the observed rate of each combination?
- Are other randomly rolled traits (barrels, magazines, masterworks, etc) affected by this?
These avenues need to be explored so we can understand the impact of this issue as players, and Bungie can investigate internally.
If necessary, I plan to continue investigating, expanding the tool to track additional weapons across different activities to see if the same trend holds. Unfortunately, my methods are crude and require large amounts of crowd-sourcing. I believe Bungie should have the tooling necessary to tackle this challenge further. My hope is that this research can shed light on the whole situation, and if an issue is discovered, that it can be resolved and players can get back to farming their favorite weapons, efficiently and fairly.
Conclusion
There is strong evidence suggesting that perk distribution issues may not be uniform. Further, this systemic problem might extend beyond just the Vesper’s Host dungeon grenade launcher, VS Chill Inhibitor. Although that weapon was my initial focus, similar patterns have emerged with other guns, like the Bitter/Sweet from the episode. Despite theoretical odds, many players are struggling to get certain perk combinations across multiple weapons, indicating this might be a broader issue with the RNG system.
To Bungie: I’ve used solid, unbiased data collection techniques to ensure the findings are reliable, and I hope Bungie will take a closer look. While I understand that Bungie has previously stated there’s no perk weighting in the loot system, the data suggests otherwise—at least for certain guns. Whether you call it perk weighting, non-random randomness, or something else, it doesn’t matter. Regardless if this is an isolated issue or a more systemic problem, I hope this research can help shed light on the situation. As a community, we would really appreciate it if Bungie could investigate further and let us know what they discover. Maybe even a dev deep dive if possible. This is a rather unprecedented occurrence. May the odds be ever in your favor.
Credits
Eko (peer review)
TStorm (peer review)
Spark peer review & for bringing this issue to life further
Vendetta & Wauby for noticing a pattern and starting the proximity theory
Everyone who used my tool to help crowd-source data
Boop for inspiration and the original Multimach data
References
- Boop’s Multimach: https://x.com/freeboop1/status/1838096821945557428
- Spark’s initial study: https://x.com/sparkd2_/status/1848142767102317000
- Light.gg Multimach: https://www.light.gg/db/items/3211624072/multimach-ccx
- Light.gg Bitter/Sweet: https://www.light.gg/db/items/2599338625/bittersweet/