7.1 "Trivial" stuff to check first:

1. Check the eventviewer. If for example the "RPC Server" is unvailable on some Node, reboot it.
   RPC is sort of the "motoroil" where Windows runs on.
2. Check the recent events as listed in the "Failover Cluster Manager".
   You Will get Hints from those events for sure.
3. Using the "Failover Cluster Manager": check on the health of all the resource groups. Are they all online?
4. Using an elevated command prompt on all Nodes: enter "cluster node".
   Get a normal response back? (that is, it reports "online nodes").
5. Ping all the Nodes on their hostnames. Get a Response back from all of them?
6. Ping the VIP's. Get a Response back from all of them?
7. All Nodes needs Permanent connectivity to AD and DNS. Is that OK?
8. Were there any changes on "service accounts of Applications" recently (which are relevant for the Cluster)?
9. Windows Firewall activated recently? Rules added or changed recently?
   Some ports MUST be open for cluster communication. See section 5.4.
10. Check the Cluster related "computer objects" in AD. Are all objects there?
11. If used: check the "witness disk"/"quorum drive" from the Node that is the owner.
12. If used: check the "witness disk"/"Fileshare". It must be visible from all nodes.
13. Check if there is a duplicate Hostname or IP on the network (should have been in the eventvwr logs).
14. Super trivial: check all connections, cables, switches.
15. Check the "C:\Windows\Cluster\ClusDB" file on all Nodes. See any "abnormal" date/time property,
    or does it all seems to be "reasonable"? Are the sizes of the files of the same "order", or are there rather large exceptions?
16. Using "regedit", take a look at the "HKLM\Cluster" hives. Is it loaded? Does it all looks "reasonable"?

7.2 Change the "QuorumArbitrationTimeMax" setting:

This section is Potentially Dangerous! Before considering using this:
crosscheck with other sources, or consult MS Support.

If you use the "Node and Disk Majority model" (used with an even number of nodes), or the "Disk only" model, and you experience
problems when the Quorum Drive owner Node changes to another Node, then there might be problems with
the Quorum Drive itself, or possibly with "Arbitration settings".

(A): The eventvwr and "Failover Cluster Manager events", should show evidence of Quorum drive troubles.
(B): The cluster log/report should show evidence of "Arbitrating" timeouts or mismatches.

If not (A) or (B) is indeed true, skip section 7.2.

It's not likely to be the cause of losing control of the Quorum Disk. However, here is some information anyway.
The "QuorumArbitrationTimeMax" and "QuorumArbitrationTimeMin" records are registry based (cluster hive).

"QuorumArbitrationTimeMax": Specifies the maximum number of seconds, that a node is allowed to spend arbitrating
for owning the quorum resource in a cluster. Default: 60 seconds.

In some very exceptional cases, it might be too low.

If MS Support instructs you to change it, you might use the "cluster" executable:

C:\TEMP\> cluster.exe /prop QuorumArbitrationTimeMax=x

(x in seconds, like for example 90)

7.3 Change the "Quorum drive":

This section is Potentially Dangerous! Before considering using this:
crosscheck with other sources, or consult MS Support.

Change the "Quorum drive": Although often proposed as a "standard procedure", no fundamental change in Clustering should be seen
as "just a routine".

If you use the "Node and Disk Majority model" (used with an even number of nodes), or the "Disk only" model, and you experience
problems with the Quorum drive, or you know that the Quorum drive is in a bad state, or there is something wrong with this LUN,
then indeed it might be needed to change the "witness disk", or the Quorum, to another (new) LUN.

(A): The eventvwr and "Failover Cluster Manager events", should show evidence of Quorum drive troubles.
(B): The cluster log/report should show evidence of Quorum drive troubles.

If not (A) or (B) is indeed true, skip section 7.3.

The general outline is like this:

1. Let the Storage Admin (or you yourself) create a LUN of at least 1GB. Perform tests on the LUN,
   like chkdsk, copying files etc.. Then format it again with NTFS.
2. Add the new LUN to your failover cluster. Make sure it's visible in "available storage" in the Failover Manager.
   
3. Perform the following in the Failover Cluster manager -> Choose More Actions -> Choose Configure quorum settings
   -> Select the correct quorum model -> Make sure the new disk is selected.
4. The cluster will reconfigure your Quorum disk. After that, the new disk will be the active Quorum disk.

7.4 Starting the Cluster when the Quorum is not met (e.g.: insufficient Nodes):

This section is Potentially Dangerous! Before considering using this:
crosscheck with other sources, or consult MS Support.

We already have seen this one before.

Please read section 1.2 first.
Then, move to section 2.1.

By design, a Cluster will not work if the number of "voters" is less than the required
minimum for a certain quorum model.
Depending on how many nodes are in the Cluster, a Cluster can survive the loss of one or more nodes.
However, if too many Nodes (and witness disk, if used) are unavailable, the Cluster shuts down.

However, if you must start the Cluster anyway, on the surviving nodes, the "/forcequorum" switch
might be considered.

Example:
In a 5 node Cluster, suppose that 3 Nodes are unavailable. But Nodes 1 & 2 are unharmed.
Then you can force the Cluster to "be up" on Nodes 1 & 2, using the following command on Node 1:

C:\TEMP> net start clussvc /forcequorum node1,node2

A powershell equivalent command exists too.

The cluster database on Node 1, is authorative, and it will be pushed to any running node,
or node that joins the Cluster again.

After Nodes have become online again, it depends on the architecture and Clustered applications,
if you still need to perform followup actions.
That's why any info in this note, has no more than a "hint status".
In general, I would advise a full shutdown, and restart again.

7.5 Starting a Cluster when no DC is available (bootstrapping):

Up to Win2K8 (inclusive), if no Domain Controller is available, you can't start a Cluster (if all was down).
To my knowledge, it just will not work.

In Win2K12, it's possible to boot a Cluster, even if no Domain Controller can be contacted.
Although you have a functional Cluster, some Administrative actions cannot work, like adding a new Node.
This is usefull in any circumstance, but especially in virtualized environments.

Section 8. SQL Server 2012 Availability Groups.

8.1 Overview:

=> Clustering, Availability Groups and AlwaysOn:

Here, you use Nodes with Win2K8 or Win2K12, where the "cluster feature" is installed on.
Then, you configure the cluster more or less in the usual way, that is, you have a number of "Nodes" and you use
a certain "quorum model".
So, you might have a 2 Node cluster, with a "disk witness" or a "file share witness". Or, you might just use
3 Nodes, without a disk or fileshare witness (see section 4).

However, from a technical standpoint, the Windows cluster is more easy to setup, since you typically do not use
"shared storage" on a SAN.
So, you can leave out the install of iSCSI or FC cards, and you do not have to configure SAN connections.
But you still need a "heartbeat" network, and a Public network which the clients use to connect to the services.

Not using a SAN may sound amazing! Maybe indeed..., but here is the "trick":

This type of cluster is Centered on SQL Server 2012/2014 Availability Group(s) (AG).
In the last phase of the cluster installation, you will work with SQL Server only.

Suppose we have a 2 Node cluster.
On node A, you create a SQL Server Instance, and create (or restore) databases on the Local storage of that Server system.
On the other node (node B) in the Cluster, you create a SQL Server Instance as well. It's important to understand that those Instances
are NOT clustered. They are just "local" instances on their own nodes. And both will be "up" all the time.

On node A you select one or more databases which needs to be placed in a "placeholder", an Availability Group, which you give
an appropriate name. It's just an abstraction which identifies those databases. Suppose you named it "Sales_AG".
On node B you tell the system that a "Replica" of "Sales_AG" needs to be installed, using the Local storage of that Server system.

Then, you select a "synchronization mechanism", which most typically will be the "synchronous-commit availability mode".
Using MS convention, on node A we have the "Primary replica", and on node B we have a "Secondary replica" (replica's of all databases in the Sales_AG).


Fig 6. AlwaysOn "Availability Groups": a "Primary AG" and it's "Replica(s)".



An availability group is the "mechanism" of a failover environment for a set of user databases.
So, those databases fail over together, as one "unit".
It's not exactly like a "traditional" SQL cluster. Here, if the Primary Node fails (where SQL runs on), then on another node,
a SQL Instance is started, and it takes over the "ownership" of Shared Storage (where the databases live), and other items like
the "virtual Service name" and the corresponding "virtual IP address". Then, after a very short while, the clients can reconnect
to that instance.

An AlwaysOn AG, consists of the (leading) Primary Replica on some Node, and one or more Secondary Replica's on other Nodes.
If the primary Node fails, the primary and the secondary role (on some preferred node) are switched,
thus providing for High Availability. So, what once was a secondary replica, now becomes the (leading) primary replica
Note that the SQL Instances on all Nodes are up and running all the time.

Also note, that the replica's usually are stored on local disks on those Servers. Ofcourse, that can be a local diskarray
using some RAID level itself. But it can also be just a local disk, without RAID.
The "protection" or "HA", sits in the fact that the replica's are distributed over 2 or more Nodes. One node has the Primary,
and other nodes have secondary replica's, which are in "sync" with the primary.
The more Nodes you use, the more "Always On" your system gets.

If the primary fails, and you have setup AlwaysOn using the "synchronous-commit availability mode", then an "automatic failover" without dataloss will occur.

A manual failover is possible too. Using the example above, where we discussed the Sales_AG Availability Group, then logon
to the secondary SQL instance (node B), and use the SQL command:

ALTER AVAILABILITY GROUP Sales_AG FAILOVER;

Then, after a short while, the primary and secondary roles are "switched", and now Node B is the leading one.

=> The listener:

Since the Primary replica, may "fail over" to another replica on another node, we need to have a "means" that
enables clients to reconnect to the new Primary Instance, without changing anything on the clients at all.

This is handled, both at a traditional Cluster, and with an Availability Group, by using a Virtual IP.
In DNS, a "Network name" is registered. This Network name represents the name to which clients connect to a SQL service.
With that "Network name", an IP address is associated. Clients just resolve the Network name to that IP, using DNS.

Now, if a new Node takes over the primary role, we only need to assure that now the new instance listens on that IP.
This is handled by the "listener". So it makes sure that the new instance that takes over the role of the Primary, assumes
that "Network name" and the associated (virtual) IP address.

To set up an "Availability Group", is really easy to do using the SQL Server Management Studio.
However, you have to have some point straight beforehand, like which databases will go into the Availability Group,
and which synchronization mode you must select during setup.
I am not going into that, but please read the following note.

Note:
If you want a few gentle introductions on how to setup a simple Cluster and "Availability Groups", you might take a look
at the following blogs/articles. The first one is very easy, and can be read in minutes. The second one (3 parts), provides for
more info, and takes a bit longer to study, but still is great stuff to read.

article 1 (blogs.technet.com).
article 2 (in 3 parts) (www.mssqltips.com).

8.2 Methods for a manual failover:

If you would take a look again at figure 4, you see the "Failover Cluster Manager", but in that figure, a traditional
Cluster was used.
Here, if I would rightclick the "sqlacc" instance, one of the choices would be "Failover to another node...".

Ofcourse, in an AlwaysOn environment you have a functional "Failover Cluster Manager" too, but you should not use it
to perform a manual failover.
That might strike you as strange indeed. You can use it, under some circumstances, but Microsoft strongly recommends using:

• the "ALTER AVAILABILITY GROUP" SQL statement, just as we have seen above.
• a Powershell statement.
  
• just simply use the graphical SQL Server Management Console (SSMS).

The reason is, that the "Failover Cluster Manager" is not exactly aware of the "synchronization status" between the replica's.

In my view, the "ALTER AVAILABILITY GROUP" is probably the easiest way. Even if you have multiple AG's, just repeat
the statement for all AG's that must failover.

8.3 SQL Server system views:

Document your cluster extensively. Use printscreens and whatever else that looks usefull.

Some SQL Server "system views" might be helpfull as well. The metadata in SQL Server, has some
interesting material too.

For example, take a look at (using "select * from ..."):

select * from sys.dm_hadr_availability_group_states
select * from sys.dm_hadr_availability_replica_states
select * from sys.dm_hadr_database_replica_states
select * from sys.dm_hadr_database_replica_cluster_states
select * from sys.dm_hadr_availability_replica_cluster_nodes
select * from sys.dm_hadr_instance_node_map
select * from sys.dm_hadr_name_id_map
select * from sys.dm_tcp_listener_states
select * from sys.availability_groups
select * from sys.availability_groups_cluster
select * from sys.availability_databases_cluster
select * from sys.dm_hadr_instance_node_map
select * from sys.availability_replicas
select * from sys.availability_group_listener_ip_addresses
select * from sys.availability_read_only_routing_lists
select * from sys.availability_group_listeners

Ofcourse, quite a few of them, can also be used to monitor, for example to view the current state of the AG's, the synchronization states,
and other "health" information.
Just try the queries, then use the queries with selections of some columns of interest. Then you might try some joins, and see
what can be usefull in your situation.

Here are a few of my own queries and joins:

(1). Which databases reside under which AG:

select substring(g.name,1,20) as "AG_Name", substring(d.database_name,1,20) as "Database"
from sys.availability_groups_cluster g, sys.availability_databases_cluster d
where g.group_id=d.group_id

(2). Which databases reside under which AG under which Instance:

select substring(g.name,1,20) as "AG_Name", substring(d.database_name,1,20) as "Database", n.instance_name
from sys.availability_groups_cluster g, sys.availability_databases_cluster d, sys.dm_hadr_instance_node_map n
where g.group_id=d.group_id AND g.resource_id=n.ag_resource_id
order by n.instance_name

(3). Checking the Health of the Replica's on an Instance:

select replica_id, substring(db_name(database_id),1,30) as "DB", is_local, synchronization_state_desc, synchronization_health_desc,
log_send_rate, log_send_queue_size
from sys.dm_hadr_database_replica_states

(4). Showing Availability Groups, and their listeners and VIP's:

select substring(g.name,1,20) as "AG name" , l.listener_id,
substring(l.dns_name,1,30) as "Listener name", l.port,
substring(l.ip_configuration_string_from_cluster,1,35) AS "VIP"
from sys.availability_groups g, sys.availability_group_listeners l
where g.group_id=l.group_id

(5). Checking if the Secondary Replica's on INST3 are "good" to perform a Fail Over. See also query (3).

select * from sys.dm_hadr_database_replica_cluster_states
where replica_id=(select replica_id from sys.availability_replicas
where replica_server_name ='INST3')

-- Other queries:

select replica_id, database_name,is_failover_ready
from sys.dm_hadr_database_replica_cluster_states

select substring(group_name,1,20) as "AG_Name",
substring(replica_server_name,1,35) as "Instance_Name",
substring(node_name,1,30) as "Node_Name"
from sys.dm_hadr_availability_replica_cluster_nodes

select substring(dns_name,1,30) as "DNS_LIstener_Name", port,
ip_configuration_string_from_cluster from sys.availability_group_listeners

select replica_id, substring(replica_server_name, 1,30) as "REPLICA_Server_Name",
substring(endpoint_url,1,30) as "Endpoint", availability_mode_desc
from sys.availability_replicas


Some of the above queries show the status of your AG's, replica's. and listeners.

8.4 Synchronization modes:

You create the primary replica, and one or more secondary replica's on other nodes.
Note that all SQL Server instances on all nodes, are "up and running" all the time (unless one fails ofcourse).

During setup, you have to choose between a few "synchronisation models". The models differ in how a transaction
at the primary is considered "to be completed".

If you require that a transaction at the primary replica, is complete, only if the secondary has received the transaction log
vectors too and applied them, then choose the "synchronous-commit availability mode". This also enables an "automatic Failover".
If a problem at the primary arises. An unattended failover is possible, since the primary and secondaries are always fully synced,
and no dataloss is guaranteed at all times. So, we can safely enable an (unattended) "automatic Failover" if it's neccessary.
However, at a high transaction rate, you must be sure that all components (like network latency) must not "spoil" things.
But, in my experience, it's almost always possible to choose the "synchronous-commit availability mode".
Also, this one is most recommended, since it enables an (unattended) "automatic Failover".

The other two modes do not enable an (unattended) "automatic Failover". So, they are much less desireable.
For example, the "High Performance" mode, means that a tranasction at the primary may complete, and the commit does not have
to "wait" for the secondaries to have completed too.
It's "High Performance" in the sense that an application only have to wait for a commit on primary replica, before a transaction
is complete, and waiting for the completion of the transaction on the secondary replica is not needed.
As said, it's lesser desireable, but if the transaction rate is high, and network bandwith is low, maybe you are forced to
choose this one.

If you want to see another one of my notes, specifically on "AlwaysOn", then use this link.

Appendix 1: articles on the internet discussing Cluster problems and solutions.

Six common problems with Failover clusters (windowsitpro.com)