Note that a majority quorum is required for the cluster to maintain write operation (and also to have a cluster leader).
If all nodes in the cluster joined and caught up successfully, you would have had a 6-node cluster. If you brought down the first 3 nodes at the same time, you now have 3/6 cluster members, you've lost quorum, so no write access and no leader.
If you instead do a gradual removal of members such that the cluster can scale down, then you can tolerate dropping down to a 3-node cluster.
Quorum is required for commits, adding, and removing members from a cluster, and member changes will alter the number of online nodes required for quorum.
Raft operations are sequential, so when there are a majority of nodes able to reach consensus on voting in or out cluster members, that implies that they are caught up on all previous commit operations.