Kaseya Performance &
Best Practices Guide
Authors: Jacques Eagle
Date: Thursday, April 29, 2010
B.0 - The Basics 5
B.1 - Hot Fixes 5
B.2 64 bit versus 32 bit OS and SQL Server. 5
B.3 - System Requirements 5
B.4 - Browsers and workstations make a difference 6
How to monitor memory 6
M.2-64 Bit Operating System 7
M.3-32 Bit /3GB in the boot.ini 7
M.4 Memory Best Practices 8
C.0 - CPU 9
C.1 - Microsoft Reporting Services 9
C.2 - KaseyaMessageSys 9
C.3 – w3wp 10
C.4 - Kaseya Add-ons 10
C.5 - Larger number of files on servers and workstations 10
C.6 - Log Archive duration 10
C.7 – IIS Compression 10
N.0 - Network 11
N.1 – IIS Compression 11
V.0 - Virtualization with ESX 12
How to monitor resources under VMWare ESX 12
V.1-VMWare ESX Memory Configuration 13
V.2-VMWare ESX CPU Configuration Reservations 14
V.3-VMWare ESX CPU Configuration Limit 17
V.4-VMWare ESX VMWare tools up to date? 17
F.0 - Federation of Servers 18
A.0 - Add-On Performance Tips 19
A.1 - Service Desk 19
Performance and Best Practices Check List 20
B -The Basics 20
M-Performance Checklist for Memory 20
C-Performance Checklist for CPU 20
V-Performance Checklist for Virtualization with ESX 21
Appendix A – Monitor Sets 22
CPU – Reporting Services 22
CPU – Kaseya Messagesys 22
CPU – w3wp IIS Workpool Process 23
CPU – SQL Server Process 23
If you are running Kaseya Version 5.x or previous versions and are close to maximizing your CPU, Networking and/or IO resources, don’t expect Version 6.x (K2) to run similarly on the same infrastructure. We hope this guide will help you with your planning on upgrading or a new install of Kaseya K2. Please remember that the Minimum Requirements on the Kaseya Support Site is based on an average. Every customer is different and your mileage will vary.
Kaseya is an enterprise application that requires specific hardware and software requirements in order to run effectively. These requirements include a database and IIS server(s), a network capable of handling agent traffic and a client platform and browser to present the user interface.
In addition, it is helpful to understand that every environment that is running Kaseya is different. There are many variables that will impact the performance of your Kaseya installation. The recommended requirements on our web site are a general guideline based on measures taken on multiple customer environments. The purpose of this document is to help you give your Kaseya Environment a Health Check by looking at various areas of performance and providing a checklist of items to review to implement best practices as it pertains to performance.
This document will go over the various areas of performance with recommendations of best practices where appropriate. Each of the sections are labeled in a way that they can be easily be referenced from the Performance Check List which is in the last section of this document. The sections covered in this guide are:
Make sure your up to date on hot fixes since performance related issues are addressed through this mechanism. Go to system, configure and make sure you have “Enable automatic check” checked. Also verify that you are current on hot fixes by looking at the “Last Hotfix” field. If not, click the “Get Lastest Hotfix” button. It is also a good idea to run the “Reapply Schema” link after hot fixes are applied since this may create any missing indexes on the database helping performance.
Some customers choose not to have this feature enabled. In this case, you should review this periodically and if you are experiencing performance issues, it’s best to apply all hot fixes before contacting support.
Don’t use 32 bit OS’s or SQL Server. It’s okay for test environments, but will not scale to support Kaseya.
Did you review the minimum system requirements at the Kaseya support site? http://www.kaseya.com/support/system-requirements.aspx
Keep in mind that these requirements can vary based on many variables such as:
It is therefore important for you to monitor your environment at all times and preferably with a Kaseya Agent. This way, you can monitor, track and alert when resources are constrained like CPU, Disk IO, Memory, SQL Server Locks, etc… Kaseya provides sample monitor sets as well as a NOC service that can monitor these for you. Please feel free to contact our NOC services for more details on this service. As we work with many clients, we constantly update our monitor sets to insure best practices in terms of monitoring the Kaseya solution.
The new features and functionality of the new Kaseya UI does require more resources in terms of Networking bandwidth and CPU on the clients workstation running the UI. The reason is that we are now using Java AJAX and JSON in order to render our pages. We have found that using Firefox over Internet Explorer provides better response times on the Kaseya UI and is highly recommended. Chrome appears to be the fastest; however, this browser is not supported at this time.
If you notice that the UI is slow, check to see how much CPU your workstation is utilizing during screen refreshes. If the CPU consistently hits 100%, your workstation may be under powered to render the pages. Also compare the CPU utilization between Firefox and Internet Explorer.
Memory can be monitored by using the resource monitor in Windows 2008. There are two key indicators that are important to memory and this is why they show on this panel. You should not see excessive Hard Faults/sec. Ideally, this number should be very low or even zero. If you see values in the double digits or more, you are running low on memory and disk swapping is occurring. This is bad. Also, make sure you are not above 85% on your Used Physical Memory. If you are, you should consider adding more RAM to your system.
You must have enough memory to support five of the most memory dependant processes in a single physical Kaseya Server environment. These processes are:
In Kaseya V6, the KaseyaMessageSys.exe is a new process that will allow Kaseya to scale with future versions and works closely with Microsoft Message Queuing. In addition to this, Version 6 also adds Microsoft Report Services.
KaseyaMessageSys can use 380 meg. of RAM for a small implementation of Kaseya (0-250 end points) to more than 1.2 Gig. Of RAM for larger installations (2500 or more end points).
In addition to KaseyaMessageSys, the new Kaseya UI requires more RAM from IIS (specifically, the IIS workpool process called w3wp). This process can use 250 Megabytes for a small implementation to 700 Megabytes or more for larger implementations.
Kaseya also utilizes Microsoft Reporting Services which will add additional memory pressure for your instances. Small instances can use 160-200 Megabytes while larger systems can use even more.
If you currently are running Kaseya V6, these memory requirements must be considered and addressed when you are ready to upgrade. Especially on systems that are running low on memory resources already. It is easy, therefore, to add another 800 Meg. to a small system just for the new Kaseya V6 release. For larger implementations, 2Gigabytes in additional RAM is not out of the question and highly recommended.
Also consider that Windows 2008 uses more RAM just to run itself. Typically, you should allocate at least 2Gig. Of RAM for the operating system alone.
Currently, it is highly recommended that even small installations of Kaseya Version 6 use 64bit Windows to address RAM over the 32 Bit limit of 4Gig. As shown above in section M.1, even a small installation of Kaseya can be impacted by new Kaseya and Windows 2008 requirements. In addition to the OS, SQL Server should be 64 Bit as well.
Kaseya has seen that the /3GB switch in the boot.ini of 32 bit Windows causes excessive paging simply because more RAM is being allocated to applications and not to the Operating system. This switch is set in the boot.ini file and should be taken out. In its place, you should upgrade your Windows operating system to 64 bit or add additional memory to your existing 32 Bit environment and using the /PAE switch in its place. PAE is used by SQL Server (by enabling AWE) and can use the additional memory above the 32 Bit limit of 4GB. This feature does not work on 32 Bit Windows Standard Edition.
If you are limited to 32 bit systems and 4Gig of RAM, you should consider splitting the server into an application and database server. This will at least allow more RAM to both the database server and the application server.
Make sure to monitor memory usage on the following processes to develop usage history and assess Memory requirements specific to your environment:
CPU utilization can vary based on many variables; however, with K2 you must plan to allocate more CPU than previous Kaseya Version 5 implementations for the same number of end points managed. This is due to the additional functionality of the back-end database and front-end. New processes, like Kaseya MessageSys and Microsoft Reporting Services will require CPU resources as well.
Microsoft Reporting Services is new to K2 and utilizes more CPU and Memory to run reports than prior versions. If you schedule many reports to run, try not to schedule them during times that the server is also running audits and patch scans. Depending on your reporting load and data volume, you can use a whole core or more just for reporting. One way to track reporting CPU utilization is to use an Agent on your Kaseya server and create a monitor set for reporting services process. From here you can create graphs as shown below using Kaseya. In this example, there is little reporting, but when reporting does run, it can take 50% or more of a core. Please note that this is not 50% of total CPU on a server. For more information, please see the Appendix section for more detailed information and the XML to import this monitor set into your own environment.
This process will allow future versions of Kaseya to scale in terms of the number of endpoints. It works closely with other Kaseya modules and internal processes. In terms of CPU, this process may use between 5-10% of a core. If you install the Kaseya Service Desk, you may find this process using more CPU. The best practice is to monitor this process as well using the monitor set described in the appendix.
This is the IIS work pool process. It should be monitored since it is second to SQL server in terms of CPU resource consumption. This process will consume more resources based on the number of administrators on your system and is also impacted by the Service Desk Add-on.
Depending on the number of add-on products like KES, BUDR, User Profile and Service Desk, your requirements can vary appreciable in terms of memory, CPU, networking and IO.
In terms of CPU, KES and Service Desk will have an impact to your overall CPU utilization. For KES, in general, you should plan on an adding an additional 10% over the recommended CPU.
The Service Desk will have an appreciable load on a server that is also running Kaseya. Currently, the following calculations should be done to estimate the additional CPU needed for single or dual server Kaseya implementation:
Just add the two above if this is a single server install. For example, if you know you will have 50 users accessing the Service Desk Add-on, you should have the following additional CPU resources added to the Kaseya Minimum requirements:
For a single Kaseya Server, just add them together. In this case, we come up with 2500Mhz. So a good rule of thumb is about 50 named users supported by one 2.4Mhz Core of CPU.
As servers and workstations become larger in size in terms of files, we must compare and store this detail on the Kaseya server. This in turn requires more space on the database and increases CPU to report and process this data especially during Audits.
The longer you keep agent and monitor logs, the more resources it takes to run reporting and data mining. Try to keep the logs to a minimum.
Some companies may consider using IIS Compression to reduce bandwidth costs at hosting centers and improving performance on low latency/slower networks. The Kaseya application will benefit from compression on IIS6.0 and IIS7.0, but there is a CPU cost. Because of the various compression levels available, it is best to create a baseline on your CPU utilization and then slowly increase compression levels until a good balance is reached with existing CPU resources and improved page performance.
Just like CPU, networking will vary based on the number of scripts run, audit scan detail, Kaseya Add-on products, etc… It is important to monitor this carefully since Kaseya Version 6 will require more network bandwidth than previous versions of Kaseya. Where this is most evident is in the Kaseya UI.
The Kaseya UI now is based on Java AJAX and JSON. These technologies bring a feature rich experience to the end user. It is very much like a windows application versus a HTML based application. These features, however, require more networking bandwidth in order to run well.
IIS Compression can be set to help with overall performance of the Kaseya UI. Compression will help with performance from the networking perspective, but will affect the CPU utilization of IIS. Compression can be set to various levels.
To enable compression, install the Static and Dynamic Content Compression role services for IIS. Once these services are installed, you must enable them in IIS by opening the IIS Manager and clicking on the Compression icon.
Then check the Enable dynamic and static Content compression on the default website as well as the IIS Home. Please note that IIS 7.0 will stop compression if the server is starving for CPU. IIS 6.0 will not. Be sure to monitor CPU utilization on the server before and after this setting to insure the impact isn’t excessive.
Customers who use ESX server should plan resources accordingly for Kaseya VM’s. In this section, we will cover best practices for Memory, IO and CPU. In some cases, you may be hosted by vendors using ESX. These recommendations should help in that situation as well.
VMWare ESX 4.0 now includes VMWare counters that help you determine the configuration and performance of your VM under ESX. These counters are installed as part of the installation of VMWare Tools for the Virtual Machine (VM). ESX 3.5 doesn’t have these counters installed as part of VMWare tools. Access to the counters is shown below. Just click on All Programs -> VMware -> WMware Tools -> start VM Statistics Logging.
From here you will see two counters related to VMWare. VM Memory and VM Processor. Each of these is used for the sections below on best practices for a VMware installation in terms of Memory and CPU.
VMware allows multiple Virtual Machines to share resources such as memory and CPU. Under certain conditions, memory can be over allocated on the ESX server, but the VM will not indicate this through its own resource monitoring. To the VM, it may think it has plenty of memory. The issue is that the ESX server may be paging memory to its own paging file impacting the performance of the underlying VM running Kaseya. To keep this from happening, insure that Virtual Machines that are running Kaseya have enough reserved memory allocated to the VM. Typically, you should reserve at a minimum 2gig of RAM for small installations and in larger, you should do 50% or more the VM’s memory. If the servers are split as an application server and a database server, you must still reserve memory to both for best performance.
As shown below, the VM Memory object has a counter called “Memory Reservation in MB”. If this is set to zero (like it is below), it is possible that the VM’s memory could be swapped to disk by the ESX host. Adjust or request that memory be reserved for your VM. Another counter called “Memory Swapped in MB” can indicate this condition. In any case, please allocate memory reservations to a single Kaseya VM or to both VMs if you have split your Database and Application Servers.
Just like memory, ESX allows VMs to share CPU. And just like memory, you need to insure that dedicated CPU cycles are reserved for single or dual server implementation of Kaseya. This is critical because inconsistent CPU will always lead to unexplained performance issues. On ESX 4.0 server VMs, use the VM Processor object to monitor CPU Performance. Here look at the CPU “Reservation in Mhz” and see if it’s set to zero as shown below. If it is, your VM can be starved of CPU cycles from the ESX host. Again, you need to allocate enough CPU reservations to satisfy the load of a single or dual Kaseya VM implementation.
How can you determine if you’re starved of CPU cycles? Use the “Effective VM Speed in Mhz” counter. This counter will show how many cycles a VM is using in Mhz. In this example, you will see that this VM has four virtual cores.
If you select the “Host Processor Speed in Mhz” counter, it will show that the host processor runs at 3000Mhz (or 3GHz). Technically, this VM if given unlimited CPU cycles from the ESX host, should have 4x3Ghz or 12Ghz total of “Effective VM Speed” (or 12,000Mhz). In the example below, you will notice that there are times that the VM’s CPU is at 100%, however, this would indicate that all 12,000 Mhz was being used to get to 100%. In reality, the VM was only getting 5077Mhz maximum as shown in the Effective VM Speed counter below. This indicates that the VM is being starved of CPU and reservations should be allocated to this server.
One reason that this server is being starved of CPU is it could be limited in how many CPU Mhz it receives. This is explained in the next section.
Even though a VM may appear to have multiple cores, there still can a limitation on how much CPU cycles are given to those cores in total. How can you tell if you are being limited in CPU? Just check the “Limit in Mhz”. This should never be below the recommended total CPU Mhz based on Kaseya minimum requirements.
Make sure you ESX VM have VMWare tools up to date. You can tell by the yellow caution in the lower right hand corner. Double click this Icon and you can run the update to bring it current. This insures that resources are properly being reported back to the ESX host.
The Service Desk requires more networking and workstation CPU resources to run. One simple way to reduce the amount of traffic and rendering at both the workstation and Kaseya server is to simply change the number of tickets that are displayed on a screen. This will have an appreciable effect on server side too since these screens are refreshed periodically. Depending on how many people you have accessing the Service Desk; the benefits can add up quickly. By default, this list is 100 per page. You should set this to 30 or less and communicate this to your users.
Service Desks greatly benefits from the browser that you’re using. We recommend Firefox over Internet Explorer.
Did you apply all hot fixes to your environment to ensure all performance hot fixes have been applied?
Did you upgrade your OS and SQL Server to 64 Bit for K2?
Did you check to make sure Minimum System Requirements are met?
If browsers are slow, are you using Firefox over Internet Explorer?
Is the workstation under powered to run Java to render the Kaseya UI?
Did you allocate additional RAM for new processes and features for Kaseya V6 if you’re upgrading from Kaseya 5?
Upgrade Windows OS and SQL to 64 Bit.
Remove /3GB from boot.ini file
Add more RAM or upgrade to windows/sql 64 Bit
Do you monitor the memory usage of key processes?
Microsoft Reporting Services - Did you create a monitor set for this process? Review this periodically and adjust CPU resources if needed. Be aware when reports are running and try not to schedule heavy scripts around this time like audits and patch scans.
KaseyaMessageSys - Did you create a monitor set for this process? Review this periodically and adjust CPU resources if needed.
W32p – Did you create a monitor set for this process? Review this periodically and adjust CPU resources if needed.
Kaseya Add –ons: Are you adding KES to your install? If yes, increase your CPU requirements 10%.
Kaseya Add-ons: Are you installing Service Desk to your install? If yes, use the following formula to calculate additional CPU requirements. Just add the two together for single Kaseya Server.
If your managed environments contain many files or programs, consider adding more CPU resources to handle the increased volume. For example, a company managing many servers in relations to workstations will see more CPU utilized during Audits.
Consider how long you need to keep logs. If you want to keep them longer than 30days (the default), then you will need to increase resources to report on those logs.
Will you be using IIS Compression to help with bandwidth and slow networks? If yes, create a CPU baseline and monitor trends as you increase compression levels on IIS. IIS 7.0 offers a CPU roll-off which will stop compressing pages as CPU utilization increases. IIS 6.0 doesn’t offer this feature, so you should insure that you monitor your CPU accordingly.
IIS Compression. If you have CPU resources, try installing and setting up compression to help with bandwidth issues. Remember, this creates more load on the CPU based on the compression you’re using.
Did you allocate reserved memory to your Kaseya implementation? If you split them into a Database and Application Server, did you still reserve memory in ESX for both?
Did you allocate reserved CPU to your Kaseya implementation? If you split them into a Database and Application Server, did you still reserve CPU in ESX for both?
Is your VM being limited in Mhz? Make sure it has enough CPU in Mhz to run Kaseya in both a single and dual VM configuration.
Is VMWare tools up to date?
Did you set the number of tickets to display to the minimum necessary?
The following monitor sets can be imported into your Kaseya Version 6.x environment by just pasting them into your monitor set import tool. Depending on your environment, you may want to change the parameters to “tune” them; however, they should provide a good base to develop your capacity models.
Kaseya also uses these monitor sets for its TI Management services as well.
These monitoring sets will help with CPU utilization of the major CPU consumption processes on a Kaseya install. Depending on the number of servers in your Kaseya environment, some of this will not apply. For example, where the database is a separate server, you would not use the SQL Server process utilization monitor set. In addition, the CPU in these monitor sets can exceed 100%. Don’t confuse this with total CPU utilization of a system. For example, a four core server with reporting services taking up two cores at 100% will read 200% in these measures. If reporting services was using all cores at 100%, you will see the measure report 400%.
These are useful in terms of trending as well as providing some feedback to support like a process that may be taking more resources suddenly.
<?xml version="1.0" encoding="ISO-8859-1" ?>
<MonitorSet name="Reporting Services Process Util" description='This set is used to monitor and alert based on reporting services process utilization'>
<Counter name='Processor Time' description='null' counterObject='Process' counter='% Processor Time' counterInstance='ReportingServicesService' counterSampleInterval='60' collectionOperator='Over' collectionThreshold='0' trendTimeSpan='1209600' trendReArm='3600' thresholdOperator='Over' thresholdAmount='50' thresholdDuration='25' thresholdWarning='10' thresholdReArm='3600'/>
<?xml version="1.0" encoding="ISO-8859-1" ?>
<MonitorSet name="KaseyaMessageSys Util" description='This set is used to monitor and alert based on Kaseya messagesys process utilization '>
<Counter name='Messagesys Processor Time' description='null' counterObject='Process' counter='% Processor Time' counterInstance='KaseyaMessageSys' counterSampleInterval='60' collectionOperator='Over' collectionThreshold='0' trendTimeSpan='1209600' trendReArm='3600' thresholdOperator='Over' thresholdAmount='98' thresholdDuration='5' thresholdWarning='10' thresholdReArm='600'/>
<?xml version="1.0" encoding="ISO-8859-1" ?>
<MonitorSet name="w3wp Process Util" description='This set is used to monitor and alert based on IIS Workpool process utilization '>
<Counter name='Processor Time' description='null' counterObject='Process' counter='% Processor Time' counterInstance='w3wp' counterSampleInterval='60' collectionOperator='Over' collectionThreshold='0' trendTimeSpan='1209600' trendReArm='3600' thresholdOperator='Over' thresholdAmount='100' thresholdDuration='25' thresholdWarning='10' thresholdReArm='3600'/>
<?xml version="1.0" encoding="ISO-8859-1" ?>
<MonitorSet name="sqlserver Process Util" description='This set is used to monitor and alert based on SQL Server Process utilization.'>
<Counter name='Sql Server Processor Time' description='null' counterObject='Process' counter='% Processor Time' counterInstance='sqlservr' counterSampleInterval='60' collectionOperator='Over' collectionThreshold='0' trendTimeSpan='1209600' trendReArm='3600' thresholdOperator='Over' thresholdAmount='300' thresholdDuration='5' thresholdWarning='10' thresholdReArm='600'/>
13 | Page