== conf-sanity test 69: replace an OST with the same index ========================================================== 10:34:38 (1789050878) start mds service on oleg241-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 10 01:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: kmemleak disabled oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg241-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg241-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg241-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 769 (time 1789050991.75 total 10.01 last 76.84) - open/close 1469 (time 1789051001.76 total 20.02 last 69.94) - open/close 2335 (time 1789051011.76 total 30.02 last 86.58) - open/close 3156 (time 1789051021.77 total 40.02 last 82.06) - open/close 4048 (time 1789051031.77 total 50.03 last 89.16) - open/close 4864 (time 1789051041.78 total 60.03 last 81.58) - open/close 5818 (time 1789051051.79 total 70.05 last 95.26) - open/close 6524 (time 1789051061.79 total 80.05 last 70.57) - open/close 7343 (time 1789051071.80 total 90.05 last 81.87) - open/close 8254 (time 1789051081.80 total 100.06 last 91.04) - open/close 8982 (time 1789051091.80 total 110.06 last 72.80) - open/close 9868 (time 1789051101.81 total 120.06 last 88.59) total: 10000 open/close in 121.50 seconds: 82.30 ops/second - unlinked 0 (time 1789051106 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 772 (time 1789051195.51 total 10.00 last 77.19) - open/close 1637 (time 1789051205.52 total 20.01 last 86.42) - open/close 2432 (time 1789051215.56 total 30.05 last 79.21) - open/close 3141 (time 1789051225.57 total 40.06 last 70.81) - open/close 3799 (time 1789051235.60 total 50.08 last 65.64) - open/close 4551 (time 1789051245.61 total 60.09 last 75.13) - open/close 5265 (time 1789051255.62 total 70.11 last 71.30) - open/close 6205 (time 1789051265.62 total 80.11 last 93.98) - open/close 6896 (time 1789051275.63 total 90.12 last 69.06) - open/close 7585 (time 1789051285.64 total 100.12 last 68.85) - open/close 8497 (time 1789051295.65 total 110.13 last 91.11) - open/close 9287 (time 1789051305.65 total 120.14 last 78.96) total: 10000 open/close in 129.95 seconds: 76.95 ops/second - unlinked 0 (time 1789051320 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 779 (time 1789051405.17 total 10.01 last 77.81) - open/close 1507 (time 1789051415.18 total 20.02 last 72.75) - open/close 2309 (time 1789051425.18 total 30.02 last 80.15) - open/close 3039 (time 1789051435.19 total 40.03 last 72.97) - open/close 3686 (time 1789051445.19 total 50.03 last 64.66) - open/close 4278 (time 1789051455.20 total 60.04 last 59.17) - open/close 5015 (time 1789051465.20 total 70.04 last 73.66) - open/close 5707 (time 1789051475.22 total 80.07 last 69.05) - open/close 6406 (time 1789051485.24 total 90.08 last 69.81) - open/close 7222 (time 1789051495.24 total 100.08 last 81.55) - open/close 8070 (time 1789051505.25 total 110.09 last 84.76) - open/close 8946 (time 1789051515.25 total 120.09 last 87.56) - open/close 9720 (time 1789051525.27 total 130.11 last 77.28) total: 10000 open/close in 133.90 seconds: 74.68 ops/second - unlinked 0 (time 1789051533 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 746 (time 1789051626.11 total 10.00 last 74.57) - open/close 1419 (time 1789051636.11 total 20.01 last 67.29) - open/close 2173 (time 1789051646.12 total 30.01 last 75.36) - open/close 2953 (time 1789051656.14 total 40.03 last 77.84) - open/close 3599 (time 1789051666.15 total 50.04 last 64.53) - open/close 4224 (time 1789051676.16 total 60.05 last 62.45) - open/close 4946 (time 1789051686.17 total 70.06 last 72.12) - open/close 5808 (time 1789051696.17 total 80.07 last 86.16) - open/close 6591 (time 1789051706.18 total 90.07 last 78.24) - open/close 7424 (time 1789051716.19 total 100.08 last 83.24) - open/close 8352 (time 1789051726.19 total 110.09 last 92.76) - open/close 9149 (time 1789051736.19 total 120.09 last 79.69) - open/close 9975 (time 1789051746.20 total 130.10 last 82.51) total: 10000 open/close in 130.38 seconds: 76.70 ops/second - unlinked 0 (time 1789051750 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 835 (time 1789051833.84 total 10.00 last 83.48) - open/close 1685 (time 1789051843.85 total 20.01 last 84.95) - open/close 2397 (time 1789051853.85 total 30.01 last 71.18) - open/close 3177 (time 1789051863.86 total 40.01 last 77.98) - open/close 3879 (time 1789051873.86 total 50.02 last 70.18) - open/close 4520 (time 1789051883.86 total 60.02 last 64.07) - open/close 5289 (time 1789051893.87 total 70.03 last 76.80) - open/close 6108 (time 1789051903.88 total 80.04 last 81.82) - open/close 6891 (time 1789051913.89 total 90.05 last 78.28) - open/close 7761 (time 1789051923.89 total 100.05 last 86.99) - open/close 8566 (time 1789051933.89 total 110.05 last 80.47) - open/close 9379 (time 1789051943.90 total 120.06 last 81.25) total: 10000 open/close in 127.86 seconds: 78.21 ops/second - unlinked 0 (time 1789051955 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 1028 (time 1789052039.30 total 10.00 last 102.76) - open/close 1697 (time 1789052049.31 total 20.01 last 66.87) - open/close 2482 (time 1789052059.32 total 30.01 last 78.45) - open/close 3279 (time 1789052069.32 total 40.02 last 79.64) - open/close 4126 (time 1789052079.33 total 50.03 last 84.64) - open/close 4909 (time 1789052089.33 total 60.03 last 78.28) - open/close 5714 (time 1789052099.34 total 70.04 last 80.47) - open/close 6530 (time 1789052109.35 total 80.05 last 81.45) - open/close 7319 (time 1789052119.36 total 90.06 last 78.88) - open/close 8113 (time 1789052129.36 total 100.06 last 79.39) - open/close 8820 (time 1789052139.37 total 110.07 last 70.60) - open/close 9714 (time 1789052149.38 total 120.08 last 89.33) total: 10000 open/close in 124.16 seconds: 80.54 ops/second - unlinked 0 (time 1789052158 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 796 (time 1789052245.95 total 10.01 last 79.55) - open/close 1655 (time 1789052255.95 total 20.01 last 85.87) - open/close 2440 (time 1789052265.96 total 30.02 last 78.43) - open/close 3238 (time 1789052275.96 total 40.02 last 79.78) - open/close 4032 (time 1789052285.97 total 50.04 last 79.28) - open/close 4918 (time 1789052295.98 total 60.04 last 88.56) - open/close 5739 (time 1789052305.98 total 70.04 last 82.09) - open/close 6668 (time 1789052315.98 total 80.04 last 92.89) - open/close 7500 (time 1789052325.98 total 90.04 last 83.18) - open/close 8206 (time 1789052336.00 total 100.06 last 70.47) - open/close 8902 (time 1789052346.01 total 110.07 last 69.59) - open/close 9615 (time 1789052356.01 total 120.07 last 71.24) total: 10000 open/close in 125.27 seconds: 79.83 ops/second - unlinked 0 (time 1789052366 ; total 1 ; last 1) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 785 (time 1789052456.74 total 10.02 last 78.38) - open/close 1583 (time 1789052466.74 total 20.02 last 79.80) - open/close 2355 (time 1789052476.75 total 30.02 last 77.17) - open/close 3132 (time 1789052486.76 total 40.04 last 77.57) - open/close 3762 (time 1789052496.77 total 50.04 last 62.98) - open/close 4402 (time 1789052506.78 total 60.05 last 63.95) - open/close 5014 (time 1789052516.78 total 70.05 last 61.18) - open/close 5596 (time 1789052526.80 total 80.07 last 58.08) - open/close 6374 (time 1789052536.81 total 90.08 last 77.74) - open/close 7312 (time 1789052546.83 total 100.10 last 93.64) - open/close 8131 (time 1789052556.83 total 110.10 last 81.89) - open/close 8885 (time 1789052566.84 total 120.11 last 75.31) - open/close 9691 (time 1789052576.85 total 130.13 last 80.49) total: 10000 open/close in 133.94 seconds: 74.66 ops/second - unlinked 0 (time 1789052584 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 832 (time 1789052668.32 total 10.00 last 83.19) - open/close 1533 (time 1789052678.33 total 20.01 last 70.03) - open/close 2269 (time 1789052688.36 total 30.04 last 73.40) - open/close 2922 (time 1789052698.36 total 40.04 last 65.26) - open/close 3676 (time 1789052708.37 total 50.05 last 75.39) - open/close 4411 (time 1789052718.37 total 60.05 last 73.47) - open/close 5309 (time 1789052728.37 total 70.05 last 89.76) - open/close 6058 (time 1789052738.38 total 80.06 last 74.88) - open/close 6990 (time 1789052748.38 total 90.06 last 93.19) - open/close 7719 (time 1789052758.40 total 100.08 last 72.70) - open/close 8332 (time 1789052768.41 total 110.08 last 61.30) - open/close 9181 (time 1789052778.41 total 120.09 last 84.87) - open/close 9959 (time 1789052788.41 total 130.09 last 77.79) total: 10000 open/close in 130.49 seconds: 76.63 ops/second - unlinked 0 (time 1789052794 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 654 (time 1789052883.25 total 10.00 last 65.37) - open/close 1579 (time 1789052893.27 total 20.02 last 92.35) - open/close 2357 (time 1789052903.27 total 30.02 last 77.79) - open/close 3072 (time 1789052913.27 total 40.02 last 71.49) - open/close 3674 (time 1789052923.28 total 50.03 last 60.13) - open/close 4456 (time 1789052933.29 total 60.04 last 78.12) - open/close 5206 (time 1789052943.30 total 70.05 last 74.93) - open/close 5890 (time 1789052953.31 total 80.06 last 68.35) - open/close 6727 (time 1789052963.32 total 90.07 last 83.61) - open/close 7502 (time 1789052973.32 total 100.08 last 77.45) - open/close 8223 (time 1789052983.34 total 110.09 last 72.01) - open/close 9133 (time 1789052993.35 total 120.10 last 90.89) - open/close 9944 (time 1789053003.36 total 130.11 last 81.02) total: 10000 open/close in 130.83 seconds: 76.43 ops/second - unlinked 0 (time 1789053009 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: kmemleak disabled modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2