Ops Checklist

	A	B	C	D	E
1
2	Ops Checklist
3	Check	System A	System B	System C	Detail description
4	Last Reviewed	Jul-15	Jul-15	Jul-15
5	Traps/Alerts
6	Trap for too many errors in webserver log	N/A	No	Yes	e.g. If more than 10 http 500 recorded in access log per minute, raise trap.
7	Trap on too many slow response in webserver log	N/A			e.g. if more than 10 http request took more than 10 seconds in a minute, raise trap.
8	Trap for ERROR, WARNING, CRITICAL, ALERT, EMERGENCY weblogic log	N/A	Yes	Yes	Hunter trap for each of these keywords. If there are already too many ERROR and WARNING, then suppress these two for a while, until we fix them. But the rest of the keyword must be in hunter trap. In IIS, there's HTTPERR log. \windows\system32\inetsrv\LogFiles
9	Trap for hogging threads	N/A	Yes	Yes	Trap on HoggingThreads >= 5. Windows: Check the perf counter for ASP.NET Requests in Application Queue
10	Trap for Queue length	N/A	Yes	No	Trap on QueueLength > 0. Windows: Check the perf counter for ASP.NET Requests in Application Queue
11	Trap on health state	N/A	Yes	Yes	Trap on Health_State != HEALTH_OK. Windows: Check App Pool enabled, Website started satus, W3SVC service running status
12	Trap on thread issues	N/A	Yes	No	Trap on Stuck threads, IdleThreads =0, Throughput >= 30, ActiveConnectionsHighCount > 30
13	Trap on files not housekeeped	N/A	No	Yes	If log files are older than 3 months, alert
14	No stdout file continuously growing and not getting rotated	N/A	No	Yes	For ex, weblogic stdout getting written to a file, which cannot be rotated, unless weblogic is restarted
15	Trap When webserver is down	N/A	Yes	Yes	Monitor web server process availability
16	Trap when important pages/service are not responding within time	N/A	No	Yes	Hit important URLs and expects response within 5 sec. If not responded or errored, raise trap
17
18
19	Database
20	Resilience
21	RAC/Mirroring	Yes	Yes	Yes	Oracle RAC or SQL Server Mirroring for all critical databases
22	All nodes evenly loaded	No	??	Yes	Check AWR to see if all nodes have more or less same Avg sessions, CPU, IO load
23	DR failover tested	No	Yes	No	Was the DR database ever brought online and DR app talked to it for a while?
24	Have up-to-date checklist for various failover scenario	No	Yes	No	You have a detail checklist, which says who needs to do what, when we go through a failover scenario. For ex, when a single DB node down, who does what. When all DB nodes are down, who does what?
25	Use TNS Service name not SID and Scan IP not Node IP everywhere	N/A	No	yes	No cron, java properties file, tns config must use Oracle SID and individual node IP. Must be Service Name and RAC cluster IP.
26
27	Storage
28	Proper storage tier	Yes	Yes	Yes	Ensure all data files, redo, archive are on Tier 2 storage. Ask Storage team.
29	High Storage latency Trap (>30ms on any tablespace)	Yes	Yes	Yes	Schedule a cron to check latency recorded by Oracle and raise trap if not within limit, SQL Server Audit records storage latency
30	Temp tablespace alert	No	Yes	Yes	Schedule cron to check temp tablespace free, if less than 40%, raise trap. SQL Server: Check tempdb max size. Current utilization.
31	Tablespace over 80% trap	Yes	Yes	Yes	Schedule cron to check tablespace being over 80%, raise trap. SQL Server: Any MDF, LDF file having <20% free space.
32	Autogrowth of tablespaces are 100MB or more	Yes	Yes	Yes	There should be no tablespace having auto growth smaller than 100MB. SQL Server: MDF, LDF must be >100MB, not a %
33	Monitor UNDO space has enough free space	Yes	Yes	Yes	Ensure UNDO space never fills up more than 80%, if it does, fix SQL or increase UNDO. SQL Server is LDF
34
35	Memory
36	SGA and PGA set as per Advisor report in last one month	Yes	Yes	Yes	Check recent AWR report and see if SGA, PGA advisory section says we need to increase SGA, PGA
37	Linux Hugepages configured properly for current SGA size	Yes	Yes	Yes	Check with Prod DBA if Oracle is using HugePages and the correct value of HugePages is configured based on SGA size.
38	ASMM configured (not ASM)	Yes	No	No	Is sga_target_size > 0
39	Trap to detect linux memory fragmentation	N/A	No	No	Is there any monitoring script which is calculating fragmented pages in Linux?
40	Trap for too low free shared pool memory	N/A	No	Yes	Cron to check free shared pool and if less than 500MB free, or too many fragmentation, raise trap.
41
42	DMBS Jobs
43	Job queries are optimized and running within expected time	No	No	Yes	Are all jobs running within hour?
44	No Job running during online hours, peak traffic	No	Yes	Yes	No expensive job is scheduled between 8AM to 6PM?
45	Jobs do not start if previous instance is already running	No	??	Yes	Job has check to ensure previous instance isn't already running.
46	Trap if job fails	No	??	No	If a job fails, email alert goes to ASG
47	Overruning job trap, exceeding max allowed time	No	??	No	If a job has exceeded 3 hours, email alert to ASG
48
49	Query
50	Application queries are optimized	No	No	Yes	e.g. No online query taking more than 5 seconds. SQL Server has Activity Monitor
51	Limit max SELECT record	No	No	No	no one can select say 10,000 records via some API call and blow up app/db
52
53	Housekeeping
54	Regular Index online rebuild for fragmented index	No	No	No	Scheduled online rebuild of important index to keep index performance good
55	Regular Table fragmentation tuning	No	No	No	Defrag highly fragmented tables regularly.
56	Large tables are partitioned	No	Yes	Yes	All multi million record table is partitioned.
57	Unused partitions are dropped	No	No	Yes	Partitions containing old records are dropped regularly via scheduled job on large tables.
58	Weekly stale statistics report	Yes	No	Yes	Report to show stale statistics last analyzed 10 days ago.
59	Weekly fragmentation report	No	No	No	Report to show heavily fragmented tables and index.
60	Weekly job run time report	No	No	No	Report to show cron and job runtime and show trend over several weeks.
61	Weekly/Month large table housekeeping	No	No	Yes	Scheduled purging of old data from large tables.
62	Purge ceased customer data	No	No	No	EU GDPR requires BT cannot hold a customer data who left 2 years ago
63	Specific stats gather job, not automatic	No	No	Yes	Do not use auto gather stats on entire database. Use specific schema gather at a time.
64	Automatic Purge logging tables	No	No	Yes	Any audit, logging type table is purged regularly via automated job.
65	Cleaned all backup/temp tables	No	No	No	No useless tables left in DB.
66	Any tablespace over 80% allocated	Yes		Yes	No tablespace over 80% allocated on DB.
67	Fixed object stats gathered monthly	No	Yes	Yes	Due to Oracle bug, fixed object stats gather is scheduled to run on a weekend, once a month.
68	Stats are locked on temp, backup tables	No	No	Yes	Useless tables have their stats locked, so that they aren't gathered automatically, wasting DB resource.
69	Are the backups running successfully without any failure	Yes		Yes	No error reported in netbackup, comvault etc. check with backup team.
70
71	Security
72	DB Links have their own user account, not using schema	Yes		Yes	DB links aren't using unlimited schema user.
73	Each client has their own user account, not using schema	Yes		No	Applications have their own user accounts and not use the privilege account, eg sa
74	ASGs have resource limit on their user account	No	No	No	ASG cannot run query that can blow up database.
75	ASGs use their own user ID and not share user account	Yes	No	Yes	All EIN/BOATID based login, not using sa/sys or any privilege account to login
76	ASGs use own ID for regular work, schema user only for deployment	Yes	No	No	Non privileged account for day to day work.
77	Latest Oracle/SQLserver version and patch	Yes	No	No	Oracle 12c or SQL Server 2018
78	Alert on schema user login	No	No	No	Email alert whenever someone logs into DB using schema user.
79
80	Misc
81	Trap to detect individual node issue, not the cluster	No	No	Yes	Every 10 mins Cron to hit each and every node individually and run a SELECT query on an important table. If the select query takes more than 1 sec, raise alert. If fails, rate trap.
82	Sequence reaching max limit trap	Yes	Yes	Yes	Every 10 mins Cron to check if sequence is over 80% of max value.
83	Trap for invalid object	Yes	No	Yes	Every 10 mins Cron to check if any object has become invalid.
84	All Nodes have same SGA, PGA etc DB parameters	Yes		Yes	Compare all DB parameters across all nodes.
85	Backup runs during out of hours	Yes		No
86	Trap when more than X sessions are blocked for more than Y minutes	No		Yes	Cron/Job to monitor for blocking sessions every 10 mins and send alert
87	Trap when one or more TNS Listener/Database is down	Yes		Yes
88
89	Crons / Windows Task
90	Cron failure monitoring	No	Yes	Yes	If a business critical cron script fails to run, raise trap.
91	Crons have exception logging	No	Yes	Yes	All business critical crons record errors in log file.
92	Detects when cron does not run on time?	No	Yes	No	A cron to check if other business critical crons have run on time.
93	Trap on overruning cron	No	Yes	No	If a cron is running longer than it should, raise trap. This requires cron to record start and stop timestamp in a log file.
94	Prevent duplicate run	No	Yes	No	Cron checks if there's another instance of same cron already running. If yes, exit.
95	Trap on 0 records processed	No		No	If cron is producing file, ensure the file has valid content. If not, raise trap.
96
97	All servers
98	Trap for high CPU, Memory using sar	No		Yes	Run cron to check if avg CPU, RAM over 80%.
99	Trap for high Disk Busy using iostat	No		Yes	Run iostat to check if any disk I/O use is over 80%, (this is not DISK SPACE)
100	Trap for high swap	No		Yes	Run cron to check if avg swap usage over 80%