Upgrading Exchange 2010 SP1 to SP2

I'm upgrading Exchange 2010 from SP1 to SP2. We currently have 6 Mailbox servers and 4 CAS servers.

The 6 mailbox servers all belong to the same DAG and copies are active:

01 <-> 02

03 <-> 04

05 <-> 06

Looking through some articles, I've seen some people stating to switch the PrimaryActiveManager to a different system when working on that particluar server. I've seen others state that you don't have to switch it.  Any ideas?

My plan is to upgrade 01,03, and 05 (I will be upgrading the CAS servers first). Once complete, failover the DBs to 01,03, and 05 and upgrade 02,04, and 06. Do I need to worry about switching the PAM? Right now it resides on 02.

Any other things I should look out for?

Thanks