由于数据分析师语句问题，导致segment好几个节点down
那么需要恢复segment
gpstop -r -M fast
此处省略一万字
20170629:14:38:30:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Stopping master standby host 192.168.102.68 mode=fast
20170629:14:38:32:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Successfully shutdown standby process on 192.168.102.68
20170629:14:38:33:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20170629:14:38:33:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-0.00% of jobs completed
20170629:14:38:43:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-12.50% of jobs completed
20170629:14:38:53:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-93.75% of jobs completed
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-100.00% of jobs completed
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-0.00% of jobs completed
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-100.00% of jobs completed
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-----------------------------------------------------
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-   Segments stopped successfully                              = 126
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-   Segments with errors during stop                           = 0
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-   
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-Segments that are currently marked down in configuration   = 2     <<<<<<<<
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-            (stop was still attempted on these segments)
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-----------------------------------------------------
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Successfully shutdown 126 of 128 segment instances 
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-No leftover gpmmon process found
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20170629:14:39:14:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20170629:14:39:14:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover shared memory
20170629:14:39:20:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Restarting System...
按正常步骤进行recovery
gprecoverseg
20170629:14:54:28:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting gprecoverseg with args: 
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.4.1 build 2'
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.4.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 10 2015 14:15:10'
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Checking if segments are ready
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:54:34:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:54:37:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Performing persistent table check
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[ERROR]:-Persistent table check gp_persistent_relation_node   <=> filesystem failed on host 192-168-102-71:50001.
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[ERROR]:-Persistent table check gp_persistent_relation_node   <=> filesystem failed on host 192-168-102-78:50000.
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[CRITICAL]:-gprecoverseg failed. (Reason='Persistent tables check failed. Please fix the persistent tables issues before running recoverseg') exiting...

查阅文档，我的版本postgres (Greenplum Database) 4.3.4.1 build 2
添加--skip-persistent-check参数再次进行recovery
gprecoverseg --skip-persistent-check

0170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting gprecoverseg with args: --skip-persistent-check
20170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.4.1 build 2'
20170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.4.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 10 2015 14:15:10'
20170629:14:59:26:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Checking if segments are ready
20170629:14:59:26:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:59:30:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Skipping persistent table check
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Greenplum instance recovery parameters
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery type              = Standard
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery 1 of 2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Synchronization mode                            = Incremental
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance host                            = 192-168-102-70
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance address                         = 10-102-103-70if2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance directory                       = /datastore_01/primary/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance port                            = 40001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance replication port                = 41001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance da directory                    = /datastore_01/primary/schemas/da/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance dm directory                    = /datastore_01/primary/schemas/dm/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance ims directory                   = /datastore_01/primary/schemas/ims/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance openreport directory            = /datastore_01/primary/schemas/openreport/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance report directory                = /datastore_01/primary/schemas/report/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance host                   = 192-168-102-71
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance address                = 10-102-104-71if3
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance directory              = /datastore_01/mirror/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance port                   = 50001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance replication port       = 51001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance da directory           = /datastore_01/mirror/schemas/da/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance dm directory           = /datastore_01/mirror/schemas/dm/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance ims directory          = /datastore_01/mirror/schemas/ims/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance openreport directory   = /datastore_01/mirror/schemas/openreport/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance report directory       = /datastore_01/mirror/schemas/report/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Target                                 = in-place
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery 2 of 2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Synchronization mode                            = Incremental
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance host                            = 192-168-102-77
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance address                         = 10-102-102-77if1
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance directory                       = /datastore_01/primary/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance port                            = 40000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance replication port                = 41000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance da directory                    = /datastore_01/primary/schemas/da/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance dm directory                    = /datastore_01/primary/schemas/dm/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance ims directory                   = /datastore_01/primary/schemas/ims/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance openreport directory            = /datastore_01/primary/schemas/openreport/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Failed instance report directory                = /datastore_01/primary/schemas/report/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance host                   = 192-168-102-78
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance address                = 10-102-103-78if2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance directory              = /datastore_01/mirror/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance port                   = 50000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance replication port       = 51000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance da directory           = /datastore_01/mirror/schemas/da/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance dm directory           = /datastore_01/mirror/schemas/dm/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance ims directory          = /datastore_01/mirror/schemas/ims/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance openreport directory   = /datastore_01/mirror/schemas/openreport/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Source instance report directory       = /datastore_01/mirror/schemas/report/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-   Recovery Target                                 = in-place
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------

Continue with segment recovery procedure Yy|Nn (default=N):
> y
20170629:15:00:15:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-2 segment(s) to recover
20170629:15:00:15:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Ensuring 2 failed segment(s) are stopped
 
20170629:15:00:16:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
updating flat files
20170629:15:00:23:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating configuration with new mirrors
20170629:15:00:24:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating mirrors
. 
20170629:15:00:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting mirrors
20170629:15:00:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
.... 
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Process results...
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating configuration to mark mirrors up
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating primaries
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary conversion of 2 segments, please wait...
................................................................................................................................................................................................................................................................................................................................................................................................................................................................ 
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Process results...
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Done updating primaries
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-******************************************************************
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating segments for resynchronization is completed.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Use  gpstate -s  to check the resynchronization progress.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-******************************************************************

最后进行role对调
gprecoverseg -r