NW Mechanics Testing Guide:

Credits:

@michela123, most of these methods were either learned or improved upon from observing the way she tests mechanics.

Note:

Although this guide and its contents are aimed towards helping players understand how to test mechanics in Neverwinter, keep in mind that you could apply similar thought processes to testing mechanics in any game, or apply them to things outside of gaming. What this guide is seeking to teach is a specific thought process, which is ultimately not an easily endeavour. I hope that throughout this document, I manage to achieve this.

Contents:

Credits:

Note:

Contents:

Required Tools:

How Advanced Combat Tracker works:

Custom Triggers and Spell Timers:

The idea behind testing:

Step 1: Identify the scope of the test.

Step 2: Make a list of everything that has the property that you are testing.

Step 3: Identify the method for testing.

Recommended Tools:

Some Tricks:

Step 4: Test it.

Step 5: Sort the results.

Step 6: Check the results.

Step 7: Bug Report.

Step 8: Presentation.

If you intend to publish your results:

Required Tools:

  1. Advanced Combat Tracker. This can be downloaded here.
  2. Neverwinter Preview installed. This lists how to get onto and how to install Preview. Whilst in theory you could test on Live, it would both be expensive and impractical, especially considering fixed damage weapons exist on preview. Both Respec tokens and Race Reroll tokens are free on preview, which makes testing many mechanics a whole lot more feasible.

How Advanced Combat Tracker works:

Although I make frequent use of this tool I by no means consider myself an expert and there is definitely a lot more that can be done with the tool then what I use it for. An explanation on how to install ACT is provided in the link above and thus is not provided here. Explanations will only be given for basic functionality.

Firstly, to run the combat log, type /combatlog 1 in game. To stop the combat log from running, type /combatlog 0. Logs are saved in Cryptic Studios\Neverwinter\Playtest\logs.

When you first run ACT, you will be confronted with a UI that looks something like this: (The numbering is my own)

main ui.PNG

  1. Main Tab. This displays whatever information it is you are importing.
  2. Options Tab. Allows you to customize ACT to your own preferences. You can add/remove some data sort methods, configure import/export methods, change the colour scheme, output to a web server (if you wish) and a whole host of other customization options.
  3. Custom Triggers Tab. See dedicated section on this.
  4. Plugins Tab. This is where you install plugins for the various games you want to test. Not much else to it.
  5. Import/Export Tab. This is where you import the combat log, to make it viewable in ACT. Alternatively, you can export an imported file from here to Excel, if you would rather use Excel to view the information.
  6. History Database Tab. ACT keeps track of what you have opened in the past. By default it keeps track of files up to 28 days ago, but you can customize that to whatever you like. If you want to go back and look at something you may have deleted, you can find it here, so long as the time period for file deletion has not passed.
  7. About Tab. Tells you about the ACT program. Nothing special.
  8. Import XML Button. I have never used it, so not certain how it works.
  9. Show Timers. It is possible to set up spell timers that count down how long till buffs expire or abilities come off cooldowns. See dedicated section.
  10. Mini Parse. Never used it, not sure how it works.
  11. Screenshot button. It takes a screenshot of either the data table, the graph, or both, depending on what you choose.

So, let us select the Import Tab:

import.PNG

  1. Import Encounters. This is where you import the log that you are interested in.
  2. Export Encounters. If you wish to export to xml or some other file type, this is where you do it.
  3. Select File. When pressed it brings up a UI that requires you to direct ACT towards the file you wish to import. The import path should look something like the path I outline at the start of this section on ACT. It should look something like this:

import pressed.PNG

Combat logs for reference is a folder I created for keeping track of whatever tests I have done, as outlined in tip number 3 above.

So, after importing a combat log, let us go back to the main tab and expand the line on the right named, “Import Zone.”

zone ui.PNG

  1. Lists all the separate encounters. Depending on what you are looking for, you may only want a specific encounter or all of them.
  2. Lists the start time of the encounter.
  3. Lists the end time of the encounter.
  4. Lists the duration of the encounter (end time - start time.)
  5. Lists the total damage dealt during the encounter by all participants.
  6. Lists the DPS of all participants during the encounter.
  7. Lists the number of participants killed.

This tab is not particularly useful, I just included this image in the event that someone is curious. Now, in this case I will select the, “all” tab (either expand it on the left or double click on it on the main table) and take a look there:

all ui.PNG

  1. The main table listing all participants and their respective information.
  2. The graph of whatever particular data type you have sorted the information by.
  3. The arrow next to damage denotes the fact that I am currently sorting the information by damage, from highest to lowest. If the arrow pointed up instead of down, it would sort from lowest to highest.

I will not explain the different tabs at the top, as they were either covered in the previous section or will be covered in the next section. Now I will select 1 of the combatants, in this case Freedom and take a look at the next UI:

  1. The different tabs Type/Damage/Etc. You can customize what is here in the options tab.
  2. Outgoing Damage. Self Explanatory, the damage that Freedom is dealing to other targets.
  3. Outgoing Healing. Lists the healing (including lifesteal) done by this character.
  4. Power Replenish (out). Lists AP gain that occurs as a result of this character. For AP gain as a percentage take the values in this tab and divide by 10.
  5. All Outgoing simply sums all the different outgoing values. Not particularly useful.
  6. Incoming Damage. Lists the damage dealt to this character from all sources.
  7. Incoming Healing. Lists health restored to this character, either by sources like lifesteal, or from heals done by other characters.
  8. Power Replenish (inc). Lists AP gained by this character, either from actions they perform, or from external sources like a DC’s gift of haste or a tact GFs Martial Mastery.
  9. All Incoming, like all outgoing, simply sums all incoming effects.
  10. The graph of the selected sorting method. It doesn’t tell you much since there is almost no correlation between these data types.

Next, let us take a look at what happens when you select 1 of these, in this case I will select outgoing damage:

  1. Effectiveness lists how effective the hit you deal actually is. A good explanation of effectiveness can be found here.

I won’t explain any of the other concepts, like, Average, Median, minhit, etc, since they are either self explanatory (in the name) or really basic concepts that you can find easily on google. If you double click on any of the individual abilities listed here, it will list all hits dealt by that ability. In this case here is disintegrate:

If you right click on any of these hits, the following piece of UI will come up:

Select Action in View logs shows you that individual hit in the logs. The advantage of this is unlike all other selection methods in ACT, this shows information about that particular hit, up to 4 decimal places. As follows (Please note all box colours overlaying this selection were added by me):

  1. The name of the attacker is selected in green.
  2. The name of the victim is selected in yellow.
  3. The name of the attack is selected in white.
  4. The damage dealt after debuffs is selected in purple.
  5. The damage dealt before debuffs is selected in red.

Selecting attacks in this manner is useful for any tests that require very precise or exact numbers.

Custom Triggers and Spell Timers:

Although not something you will find yourself using very often when testing something, the Custom Spell Triggers are great in actual gameplay for monitoring which buffs are up. First of all, in order to make use of Custom Triggers, you need to be live parsing. To do this go to options, Miscellaneous:

live parse.PNG

Now select the combat log you wish to parse. In order for this to work, you will also need to scroll down the options to Neverwinter - General and add player detection, for example:

player detection.PNG

These will be the players that ACT will actively track in combat. Now, switch to the custom trigger tab, you should see the following:

Custom Triggers.PNG

  1. When adding an ability, you enter the ability name, as it appears in the combat log, over here for it to be added. For example, to add Divine Glow in Divinity mode, you type, “Divine Glow - Divinity” in order for it to be added.
  2. This is what you will refer to the Custom Trigger as.
  3. Pressing this button will add a new Trigger with the properties you just gave it.
  4. Removes the selected Trigger.
  5. This is the list of Custom Triggers you have.

For the sake of demonstration, let us add a trigger called, “Chaotic Fury” and give it a Tab name of CMF:

demonstration.PNG

Now, click on the button “Show Timers” and then right click on the menu that appears, the following menu should appear:

spell timer options.PNG

  1. Lists currently existing timers.
  2. Over here enter the name of the Custom Trigger you created. In the case of the example above, you would enter, “CMF.”
  3. Enter the duration of the Custom Trigger, for the sake of the example, the duration of the Chaotic Fury buff is 10 seconds.
  4. Enter at which point you want to be warned that the buff will expire. This is optional, I like to set this to a value of 2-3 seconds, so 2-3 seconds before the buff expires it warns me.
  5. This is when it will remove the timer from the Spell Timer UI. I like to set this to 0.
  6. If you wish to whitelist specific players only.
  7. Where you whitelist specific players.
  8. This determines where the Spell Timer will display, Panel A is the default Panel, however you could also display in Panel B if you so desire, which you will then have to enable.
  9. These are the options for the 2 Spell Timer Panels. I recommend turning down Panel Opacity to a low value so you can see through it and make sure it is always set to being on top. I also recommend lowering the scale factor of the icons a lot, as they are quite large by default.
  10. This is the sound it will play when the trigger is activated. To select various sounds, click on the 3 ellipses dots on the left.
  11. This is the sound that will play when it is warning you the trigger is about to expire.

Spell Timers can be very useful in coordinated groups for ensuring all buffs are up before landing big burst hits. They can also be useful for optimizing your rotation, as you can build your rotation around keeping buffs up at all times.

The idea behind testing:

With that (admittedly basic) explanation of how ACT works, we can now move on to actually looking at how to test something in an accurate manner. When testing mechanics,

Step 1: Identify the scope of the test.

The first thing you want to do is determine how large (or small) the test you are trying to do is. For example, are you testing something small, like whether or not the yeti active bonus works, or are you testing something large, like how every debuff in the game works. It may seem like a silly thing to do, but depending on how large a test is, you may want to approach it in a different manner. For small scale tests, you can skip most of this except step 1, 3, 4 and 6, for large scale tests, you will want to go through all of it.

In the test I did for the ability coefficients for skills, the scope was fairly obviously large, as I was checking a mechanic across every class.

Step 2: Make a list of everything that has the property that you are testing.

Before you actually start testing a mechanic, you will want to list everything that has that mechanic. For example, if you are testing damage reduction debuffs, the first thing you would want to do is go through the list of all abilities for every class and write down every ability that describes itself as having some debuff. Once you have done this, you then move on to items. The wiki is your best friend here, for example, this page on artifacts, or this page on companions.

Once you have a list, you will firstly have a better idea of what you have got yourself into and secondly, you will be able to organize best how you plan to work through it. For large tests I recommend breaking the test down into smaller ones, where you test maybe test 1 class on 1 day and then another on another day, it simply makes the test easier to manage.

For testing the ability coefficients of skills, I made this list, keep in mind this is the final document so it looks a lot better than it did while I was testing stuff since I took some time to format it.

Step 3: Identify the method for testing.

The first thing you need to do when you set out to test something is to come up with a method for the test. The method needs to fulfill the following criteria:

  1. It needs to be rigorous. What this means is there needs to be no margin of error, or as little margin of error as possible. The best way to achieve this is to eliminate as many variables as possible. Testing with as few feats, boons, companions, insignia bonuses, gear pieces, etc as possible is ideal. If testing something related to damage or healing, you pretty much have to test using fixed damage weapons.
  2. It needs to be replicable. If your test cannot be replicated, then there is no way to verify the authenticity of the results.
  3. Try to keep the method as simple as possible. The more steps you introduce, the more likely you are to make a mistake.

When you come up with your method, you should ask yourself, “How does this mechanic interact with stuff in a way that I can check it scientifically.”

Here are some examples of methods for testing some mechanics:

  1. If you are testing a buff, you can use fixed damage weapons on preview to control the amount of damage dealt on each hit. You can get a few hits without the buff, with no other variables changing and then a few hits with the buff.
  2. If you are testing a DR Debuff, effectiveness is registered in ACT. By first eliminating all other debuffs and then applying the debuff, you can take note of the change in effectiveness.
  3. If you are testing how the power stat works, you could equip fixed damage weapons and then increase your power in increments and observe how the damage dealt by a specific ability changes. With enough pieces of information, you could then accurately model the stat curve for power.

Recommended Tools:

Here is a list of tools I recommend taking advantage of when testing:

  1. I recommend you have a guild of your own on preview, you may need to get 4 other people help you set it up. This helps you to test without other people disturbing you and ruining your tests.
  2. I recommend you have ~10 of each lesser, normal and greater mark set aside on live. You can copy these across whenever you need to upgrade artifacts etc on preview for testing purposes.
  3. Bells can be helpful for summoning Dragons on the SH map to test some mechanics, assuming you can get enough accounts (5) together on 1 SH map on preview to summon them. Alternatively, Dragons spawn at 45 minutes past the hour in Well of Dragons.
  4. For RP, you can either use the preview merchant in Bryn Shandar who sells the Eye of the Giant or you can use the Transcendent Enchantments sold in the preview Wondrous Bazaar. You can also use the artifact set from the merchant in Bryn Shandar to upgrade artifact gear.
  5. Stockpile Tomes of Experience from overflow rewards. You can use these to quickly get excess skill points on toons that need them on preview.
  6. The Wondrous Bazaar sells Fixed Damage Weapons on preview. For testing any mechanic that relies on Weapon Damage these are ideal.

Some Tricks:

  1. There are some monsters that exist that deal a fixed amount of damage and can be used to test some mechanics in PVE. @michela123 outlines these in this thread.
  2. If you are testing something that binds when you use it, I recommend renaming the character you bind it to on preview to the name of the item. This way, if you ever need to test it again later, it is easy to find. I am too lazy to do this (and always regret it later) but it works well.
  3. I recommend after you finished testing something turning off the combat log and renaming the log file to whatever it is you have tested. I keep them all stored in a folder so that if I ever need to refer back to preview tests I still have the logs.

To illustrate this method, let us go through the process with a test I have already completed, which was the test for the ability coefficients for skills. The method I used is outlined here in the, “how to test this,” section.

Step 4: Test it.

Begin applying the method you came up with in step 1 to everything you are testing. Here is some general advice for recording results:

  1. If something is, “close to what you expect,” but not, “what you expect,” don’t fall for the trap of writing down what you expect. Write down the actual result. You are trying to record how stuff behaves, not how you want it to behave.
  2. If you are testing against a dummy and the item/ability in question isn’t working, or doesn’t work as described in the tooltip, then also check it against monsters. Dummies have some bugs, so ensure it is not just a dummy bug.
  3. Make notes if you notice anything interesting, even if it doesn’t necessarily relate directly to the mechanic you are testing. You may find that information useful later.

Step 5: Sort the results.

Anything that does work the way it is intended, can now be ignored. I like to mark it in green in my spreadsheets when i am still in the testing process, to indicate that it is ok, but any colour (or no colour) will do. Now that you have a list of everything that does NOT work the way you expected it to, you can now try and figure out how it actually works. There is no best way to do this, the way I do it is I play a guessing game, for example guessing if it add or multiplies with different things, until my guessed behavior matches the actual behavior.

Step 6: Check the results.

Now that you have a list of how you expect everything to work, check it, at the very least 1 time, ideally 2 or 3 times. Everyone makes mistakes and the more complicated a test is, the more mistakes you are likely to make. By checking results, you can find these errors and then try and eliminate them.

Step 7: Bug Report.

Report everything that doesn’t behave the way it is intended to behave. Try to be as concise as possible and make sure to list the method you used to test it, as a method that could be used to replicate the results. List the actual behavior compared to the behavior described on the tooltip.

Step 8: Presentation.

This may not seem important, but making sure your test is presented in an easy to read, sensible manner is necessary, even if you are the only one who ever sees it. At the very least, it needs to be presented in a manner that makes sense to you. If your results are a pain to read, then it is easy to make mistakes if you ever need to retest due to changes or other reasons.

If you intend to publish your results:

Firstly, thanks, we all appreciate the work you have put into testing whatever it is you have tested, however, here are a few guidelines and requests from me:

  1. Don’t falsify results. It is really frustrating to see false information get spread.
  2. Expect criticism, it will come. Also, expect someone to find that 1 mistake you missed, check to see if they are correct and if so, update your results to reflect this and give credit where due.
  3. Try to keep your tests up to date.
  4. Explain your method, so that if anyone wants to check your test, they can.