== conf-sanity test 69: replace an OST with the same index ========================================================== 09:14:07 (1789996447) start mds service on oleg430-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940192 Sep 21 05:46 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: kmemleak disabled oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg430-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg430-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg430-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 861 (time 1789996540.85 total 10.02 last 85.96) - open/close 1779 (time 1789996550.86 total 20.02 last 91.74) - open/close 2686 (time 1789996560.86 total 30.03 last 90.67) - open/close 3612 (time 1789996570.87 total 40.03 last 92.54) - open/close 4475 (time 1789996580.87 total 50.04 last 86.27) - open/close 5368 (time 1789996590.89 total 60.05 last 89.19) - open/close 6263 (time 1789996600.91 total 70.07 last 89.32) - open/close 7077 (time 1789996610.91 total 80.07 last 81.39) - open/close 7953 (time 1789996620.92 total 90.08 last 87.49) - open/close 8820 (time 1789996630.93 total 100.09 last 86.63) - open/close 9685 (time 1789996640.93 total 110.09 last 86.49) total: 10000 open/close in 113.67 seconds: 87.97 ops/second - unlinked 0 (time 1789996648 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 907 (time 1789996724.11 total 10.01 last 90.63) - open/close 1840 (time 1789996734.12 total 20.02 last 93.22) - open/close 2767 (time 1789996744.14 total 30.03 last 92.55) - open/close 3645 (time 1789996754.14 total 40.04 last 87.73) - open/close 4522 (time 1789996764.15 total 50.04 last 87.69) - open/close 5380 (time 1789996774.15 total 60.04 last 85.77) - open/close 6263 (time 1789996784.15 total 70.05 last 88.26) - open/close 7205 (time 1789996794.16 total 80.05 last 94.17) - open/close 8110 (time 1789996804.17 total 90.06 last 90.39) - open/close 8976 (time 1789996814.18 total 100.07 last 86.54) - open/close 9898 (time 1789996824.18 total 110.08 last 92.14) total: 10000 open/close in 111.15 seconds: 89.97 ops/second - unlinked 0 (time 1789996829 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 793 (time 1789996913.24 total 10.02 last 79.12) - open/close 1595 (time 1789996923.25 total 20.03 last 80.14) - open/close 2414 (time 1789996933.28 total 30.06 last 81.65) - open/close 3279 (time 1789996943.28 total 40.07 last 86.45) - open/close 4157 (time 1789996953.29 total 50.07 last 87.74) - open/close 5019 (time 1789996963.30 total 60.08 last 86.14) - open/close 5861 (time 1789996973.31 total 70.09 last 84.14) - open/close 6754 (time 1789996983.31 total 80.10 last 89.22) - open/close 7625 (time 1789996993.32 total 90.10 last 87.04) - open/close 8514 (time 1789997003.33 total 100.11 last 88.80) - open/close 9408 (time 1789997013.34 total 110.12 last 89.35) total: 10000 open/close in 117.19 seconds: 85.33 ops/second - unlinked 0 (time 1789997024 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 897 (time 1789997102.84 total 10.01 last 89.61) - open/close 1717 (time 1789997112.85 total 20.03 last 81.86) - open/close 2595 (time 1789997122.85 total 30.03 last 87.79) - open/close 3499 (time 1789997132.86 total 40.03 last 90.38) - open/close 4377 (time 1789997142.86 total 50.03 last 87.77) - open/close 5315 (time 1789997152.87 total 60.04 last 93.73) - open/close 6201 (time 1789997162.88 total 70.05 last 88.50) - open/close 7085 (time 1789997172.89 total 80.06 last 88.33) - open/close 7990 (time 1789997182.89 total 90.07 last 90.45) - open/close 8899 (time 1789997192.90 total 100.07 last 90.85) - open/close 9846 (time 1789997202.90 total 110.07 last 94.69) total: 10000 open/close in 111.76 seconds: 89.48 ops/second - unlinked 0 (time 1789997207 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 904 (time 1789997280.70 total 10.02 last 90.26) - open/close 1803 (time 1789997290.71 total 20.02 last 89.81) - open/close 2745 (time 1789997300.71 total 30.03 last 94.19) - open/close 3645 (time 1789997310.71 total 40.03 last 89.96) - open/close 4565 (time 1789997320.73 total 50.04 last 91.89) - open/close 5470 (time 1789997330.73 total 60.05 last 90.47) - open/close 6411 (time 1789997340.73 total 70.05 last 94.08) - open/close 7339 (time 1789997350.74 total 80.06 last 92.72) - open/close 8357 (time 1789997360.74 total 90.06 last 101.78) - open/close 9411 (time 1789997370.74 total 100.06 last 105.37) total: 10000 open/close in 105.44 seconds: 94.84 ops/second - unlinked 0 (time 1789997379 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1171 (time 1789997445.05 total 10.00 last 117.08) - open/close 2358 (time 1789997455.05 total 20.00 last 118.68) - open/close 3525 (time 1789997465.06 total 30.01 last 116.63) - open/close 4717 (time 1789997475.06 total 40.01 last 119.18) - open/close 5967 (time 1789997485.06 total 50.01 last 124.95) - open/close 7082 (time 1789997495.07 total 60.02 last 111.49) - open/close 8322 (time 1789997505.07 total 70.02 last 123.97) - open/close 9561 (time 1789997515.07 total 80.02 last 123.83) total: 10000 open/close in 84.70 seconds: 118.06 ops/second - unlinked 0 (time 1789997522 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1144 (time 1789997586.01 total 10.00 last 114.36) - open/close 2151 (time 1789997596.01 total 20.00 last 100.69) - open/close 3167 (time 1789997606.01 total 30.01 last 101.58) - open/close 4198 (time 1789997616.01 total 40.01 last 103.09) - open/close 5212 (time 1789997626.02 total 50.02 last 101.32) - open/close 6183 (time 1789997636.02 total 60.02 last 97.09) - open/close 7113 (time 1789997646.02 total 70.02 last 93.00) - open/close 8147 (time 1789997656.03 total 80.03 last 103.30) - open/close 9124 (time 1789997666.03 total 90.03 last 97.66) total: 10000 open/close in 99.81 seconds: 100.19 ops/second - unlinked 0 (time 1789997681 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 892 (time 1789997767.92 total 10.01 last 89.13) - open/close 1822 (time 1789997777.92 total 20.01 last 92.98) - open/close 2744 (time 1789997787.93 total 30.02 last 92.15) - open/close 3646 (time 1789997797.93 total 40.02 last 90.18) - open/close 4596 (time 1789997807.93 total 50.02 last 94.98) - open/close 5529 (time 1789997817.95 total 60.03 last 93.17) - open/close 6577 (time 1789997827.95 total 70.04 last 104.77) - open/close 7530 (time 1789997837.95 total 80.04 last 95.30) - open/close 8439 (time 1789997847.96 total 90.05 last 90.81) - open/close 9331 (time 1789997857.97 total 100.05 last 89.13) total: 10000 open/close in 107.40 seconds: 93.11 ops/second - unlinked 0 (time 1789997868 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 814 (time 1789997943.37 total 10.00 last 81.40) - open/close 1721 (time 1789997953.38 total 20.00 last 90.68) - open/close 2693 (time 1789997963.39 total 30.02 last 97.06) - open/close 3659 (time 1789997973.40 total 40.02 last 96.55) - open/close 4662 (time 1789997983.40 total 50.03 last 100.26) - open/close 5682 (time 1789997993.41 total 60.03 last 101.95) - open/close 6650 (time 1789998003.41 total 70.04 last 96.75) - open/close 7769 (time 1789998013.41 total 80.04 last 111.88) - open/close 8834 (time 1789998023.41 total 90.04 last 106.49) - open/close 9847 (time 1789998033.41 total 100.04 last 101.29) total: 10000 open/close in 101.37 seconds: 98.65 ops/second - unlinked 0 (time 1789998037 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 860 (time 1789998110.61 total 10.01 last 85.91) - open/close 1855 (time 1789998120.63 total 20.02 last 99.36) - open/close 2715 (time 1789998130.63 total 30.03 last 85.97) - open/close 3546 (time 1789998140.63 total 40.03 last 83.08) - open/close 4506 (time 1789998150.64 total 50.03 last 95.97) - open/close 5409 (time 1789998160.64 total 60.04 last 90.23) - open/close 6293 (time 1789998170.65 total 70.04 last 88.37) - open/close 7168 (time 1789998180.65 total 80.05 last 87.43) - open/close 8223 (time 1789998190.66 total 90.06 last 105.42) - open/close 9460 (time 1789998200.67 total 100.07 last 123.62) total: 10000 open/close in 104.46 seconds: 95.73 ops/second - unlinked 0 (time 1789998207 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.130@tcp sys.timeout=20 start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: kmemleak disabled modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2