== conf-sanity test 69: replace an OST with the same index ========================================================== 04:05:14 (1788941114) start mds service on oleg641-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 8 20:11 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing load_modules_local oleg641-server: kmemleak disabled oleg641-server: Loading modules from /home/green/git/lustre-release/lustre oleg641-server: detected 4 online CPUs by sysfs oleg641-server: MODOPTS_LIBCFS= oleg641-server: Force libcfs to create 2 CPU partitions oleg641-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg641-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg641-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg641-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg641-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 694 (time 1788941227.62 total 10.01 last 69.30) - open/close 1432 (time 1788941237.63 total 20.02 last 73.77) - open/close 2207 (time 1788941247.63 total 30.02 last 77.49) - open/close 2922 (time 1788941257.63 total 40.02 last 71.48) - open/close 3675 (time 1788941267.64 total 50.03 last 75.24) - open/close 4443 (time 1788941277.68 total 60.07 last 76.51) - open/close 5306 (time 1788941287.69 total 70.08 last 86.21) - open/close 6125 (time 1788941297.69 total 80.09 last 81.84) - open/close 6928 (time 1788941307.69 total 90.09 last 80.30) - open/close 7791 (time 1788941317.69 total 100.09 last 86.30) - open/close 8643 (time 1788941327.70 total 110.09 last 85.20) - open/close 9367 (time 1788941337.70 total 120.10 last 72.35) total: 10000 open/close in 129.61 seconds: 77.15 ops/second - unlinked 0 (time 1788941351 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 692 (time 1788941451.78 total 10.00 last 69.20) - open/close 1447 (time 1788941461.78 total 20.00 last 75.49) - open/close 2206 (time 1788941471.78 total 30.01 last 75.88) - open/close 3057 (time 1788941481.79 total 40.01 last 85.08) - open/close 3890 (time 1788941491.79 total 50.01 last 83.26) - open/close 4715 (time 1788941501.79 total 60.01 last 82.48) - open/close 5496 (time 1788941511.80 total 70.02 last 78.09) - open/close 6160 (time 1788941521.80 total 80.02 last 66.40) - open/close 6754 (time 1788941531.81 total 90.03 last 59.33) - open/close 7480 (time 1788941541.81 total 100.03 last 72.57) - open/close 8166 (time 1788941551.82 total 110.04 last 68.54) - open/close 8741 (time 1788941561.83 total 120.05 last 57.43) - open/close 9311 (time 1788941571.85 total 130.07 last 56.91) - open/close 9803 (time 1788941581.85 total 140.07 last 49.18) total: 10000 open/close in 143.08 seconds: 69.89 ops/second - unlinked 0 (time 1788941591 ; total 1 ; last 1) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 826 (time 1788941673.26 total 10.02 last 82.46) - open/close 1587 (time 1788941683.26 total 20.02 last 76.08) - open/close 2448 (time 1788941693.27 total 30.03 last 86.02) - open/close 3249 (time 1788941703.28 total 40.04 last 80.01) - open/close 4039 (time 1788941713.29 total 50.05 last 78.96) - open/close 4903 (time 1788941723.29 total 60.05 last 86.34) - open/close 5731 (time 1788941733.31 total 70.06 last 82.71) - open/close 6533 (time 1788941743.31 total 80.07 last 80.18) - open/close 7338 (time 1788941753.31 total 90.07 last 80.45) - open/close 8124 (time 1788941763.33 total 100.08 last 78.50) - open/close 8946 (time 1788941773.33 total 110.09 last 82.20) - open/close 9745 (time 1788941783.33 total 120.09 last 79.88) total: 10000 open/close in 123.08 seconds: 81.24 ops/second - unlinked 0 (time 1788941790 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 655 (time 1788941873.67 total 10.02 last 65.34) - open/close 1227 (time 1788941883.68 total 20.03 last 57.15) - open/close 1977 (time 1788941893.69 total 30.05 last 74.88) - open/close 2740 (time 1788941903.70 total 40.06 last 76.24) - open/close 3487 (time 1788941913.70 total 50.06 last 74.68) - open/close 4313 (time 1788941923.71 total 60.07 last 82.51) - open/close 5178 (time 1788941933.72 total 70.08 last 86.43) - open/close 5890 (time 1788941943.73 total 80.09 last 71.13) - open/close 6719 (time 1788941953.73 total 90.09 last 82.87) - open/close 7620 (time 1788941963.74 total 100.10 last 90.05) - open/close 8511 (time 1788941973.74 total 110.10 last 89.07) - open/close 9309 (time 1788941983.75 total 120.11 last 79.73) total: 10000 open/close in 129.03 seconds: 77.50 ops/second - unlinked 0 (time 1788941997 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 613 (time 1788942080.26 total 10.01 last 61.25) - open/close 1383 (time 1788942090.26 total 20.01 last 76.99) - open/close 2151 (time 1788942100.26 total 30.01 last 76.78) - open/close 2897 (time 1788942110.28 total 40.03 last 74.45) - open/close 3719 (time 1788942120.28 total 50.03 last 82.17) - open/close 4556 (time 1788942130.28 total 60.03 last 83.70) - open/close 5364 (time 1788942140.32 total 70.07 last 80.54) - open/close 6227 (time 1788942150.33 total 80.08 last 86.21) - open/close 7013 (time 1788942160.33 total 90.08 last 78.59) - open/close 7707 (time 1788942170.34 total 100.09 last 69.35) - open/close 8495 (time 1788942180.34 total 110.09 last 78.74) - open/close 9230 (time 1788942190.35 total 120.10 last 73.47) total: 10000 open/close in 129.12 seconds: 77.45 ops/second - unlinked 0 (time 1788942203 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 772 (time 1788942286.95 total 10.01 last 77.13) - open/close 1593 (time 1788942296.96 total 20.01 last 82.08) - open/close 2445 (time 1788942306.96 total 30.02 last 85.12) - open/close 3321 (time 1788942316.97 total 40.03 last 87.54) - open/close 4116 (time 1788942326.98 total 50.04 last 79.41) - open/close 4939 (time 1788942336.99 total 60.04 last 82.26) - open/close 5692 (time 1788942346.99 total 70.04 last 75.30) - open/close 6526 (time 1788942356.99 total 80.05 last 83.39) - open/close 7382 (time 1788942367.01 total 90.07 last 85.41) - open/close 8266 (time 1788942377.02 total 100.08 last 88.33) - open/close 9172 (time 1788942387.03 total 110.09 last 90.51) total: 10000 open/close in 118.91 seconds: 84.09 ops/second - unlinked 0 (time 1788942399 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 786 (time 1788942480.16 total 10.01 last 78.53) - open/close 1645 (time 1788942490.16 total 20.01 last 85.87) - open/close 2468 (time 1788942500.18 total 30.02 last 82.21) - open/close 3161 (time 1788942510.19 total 40.04 last 69.21) - open/close 3685 (time 1788942520.19 total 50.04 last 52.38) - open/close 4266 (time 1788942530.20 total 60.05 last 58.06) - open/close 4965 (time 1788942540.20 total 70.05 last 69.89) - open/close 5689 (time 1788942550.22 total 80.07 last 72.28) - open/close 6447 (time 1788942560.22 total 90.07 last 75.77) - open/close 7363 (time 1788942570.23 total 100.08 last 91.53) - open/close 8413 (time 1788942580.23 total 110.08 last 104.97) - open/close 9311 (time 1788942590.24 total 120.09 last 89.76) total: 10000 open/close in 128.16 seconds: 78.03 ops/second - unlinked 0 (time 1788942602 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 784 (time 1788942691.55 total 10.02 last 78.25) - open/close 1530 (time 1788942701.55 total 20.02 last 74.60) - open/close 2374 (time 1788942711.56 total 30.03 last 84.28) - open/close 3178 (time 1788942721.57 total 40.04 last 80.37) - open/close 3859 (time 1788942731.57 total 50.04 last 68.08) - open/close 4599 (time 1788942741.58 total 60.05 last 73.96) - open/close 5489 (time 1788942751.60 total 70.07 last 88.78) - open/close 6371 (time 1788942761.61 total 80.07 last 88.16) - open/close 7222 (time 1788942771.63 total 90.10 last 84.87) - open/close 8030 (time 1788942781.65 total 100.12 last 80.66) - open/close 8830 (time 1788942791.67 total 110.14 last 79.80) - open/close 9677 (time 1788942801.68 total 120.15 last 84.63) total: 10000 open/close in 123.81 seconds: 80.77 ops/second - unlinked 0 (time 1788942808 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 873 (time 1788942887.29 total 10.01 last 87.25) - open/close 1713 (time 1788942897.30 total 20.01 last 83.92) - open/close 2587 (time 1788942907.31 total 30.02 last 87.35) - open/close 3440 (time 1788942917.31 total 40.02 last 85.29) - open/close 4280 (time 1788942927.32 total 50.03 last 83.95) - open/close 5158 (time 1788942937.33 total 60.04 last 87.71) - open/close 6008 (time 1788942947.33 total 70.04 last 84.96) - open/close 6907 (time 1788942957.34 total 80.05 last 89.80) - open/close 7680 (time 1788942967.34 total 90.05 last 77.29) - open/close 8479 (time 1788942977.35 total 100.06 last 79.88) - open/close 9362 (time 1788942987.35 total 110.06 last 88.27) total: 10000 open/close in 117.76 seconds: 84.92 ops/second - unlinked 0 (time 1788942998 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 831 (time 1788943080.84 total 10.01 last 83.04) - open/close 1781 (time 1788943090.84 total 20.01 last 94.94) - open/close 2593 (time 1788943100.84 total 30.01 last 81.19) - open/close 3473 (time 1788943110.85 total 40.02 last 87.92) - open/close 4369 (time 1788943120.86 total 50.03 last 89.50) - open/close 5227 (time 1788943130.88 total 60.05 last 85.69) - open/close 5968 (time 1788943140.89 total 70.06 last 74.03) - open/close 6719 (time 1788943150.90 total 80.07 last 75.00) - open/close 7424 (time 1788943160.91 total 90.08 last 70.44) - open/close 8236 (time 1788943170.92 total 100.09 last 81.13) - open/close 9065 (time 1788943180.93 total 110.10 last 82.75) - open/close 9852 (time 1788943190.94 total 120.11 last 78.69) total: 10000 open/close in 122.06 seconds: 81.93 ops/second - unlinked 0 (time 1788943197 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg641-server: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg641-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg641-server: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg641-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.141@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.141@tcp sys.timeout=20 start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg641-server: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg641-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg641-server: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg641-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg641-server unloading modules via unload_modules_local on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing unload_modules_local oleg641-server: kmemleak disabled modules unloaded. pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2