== conf-sanity test 69: replace an OST with the same index ========================================================== 17:11:44 (1788642704) start mds service on oleg605-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing load_modules_local oleg605-server: kmemleak disabled oleg605-server: Loading modules from /home/green/git/lustre-release/lustre oleg605-server: detected 4 online CPUs by sysfs oleg605-server: MODOPTS_LIBCFS= oleg605-server: Force libcfs to create 2 CPU partitions oleg605-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg605-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg605-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg605-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg605-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 978 (time 1788642811.21 total 10.00 last 97.77) - open/close 1936 (time 1788642821.22 total 20.02 last 95.65) - open/close 2893 (time 1788642831.24 total 30.03 last 95.58) - open/close 3788 (time 1788642841.24 total 40.04 last 89.44) - open/close 4716 (time 1788642851.26 total 50.05 last 92.67) - open/close 5674 (time 1788642861.26 total 60.05 last 95.78) - open/close 6582 (time 1788642871.27 total 70.06 last 90.73) - open/close 7580 (time 1788642881.27 total 80.06 last 99.77) - open/close 8615 (time 1788642891.28 total 90.08 last 103.35) - open/close 9641 (time 1788642901.29 total 100.08 last 102.57) total: 10000 open/close in 103.59 seconds: 96.53 ops/second - unlinked 0 (time 1788642908 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 991 (time 1788642982.88 total 10.00 last 99.08) - open/close 1915 (time 1788642992.89 total 20.01 last 92.36) - open/close 2912 (time 1788643002.89 total 30.01 last 99.68) - open/close 3948 (time 1788643012.90 total 40.02 last 103.51) - open/close 4947 (time 1788643022.91 total 50.03 last 99.79) - open/close 5980 (time 1788643032.91 total 60.03 last 103.28) - open/close 6943 (time 1788643042.91 total 70.03 last 96.27) - open/close 7922 (time 1788643052.92 total 80.04 last 97.79) - open/close 8885 (time 1788643062.95 total 90.07 last 96.06) - open/close 9841 (time 1788643072.95 total 100.08 last 95.54) total: 10000 open/close in 101.86 seconds: 98.17 ops/second - unlinked 0 (time 1788643078 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 917 (time 1788643151.33 total 10.01 last 91.64) - open/close 1805 (time 1788643161.33 total 20.01 last 88.80) - open/close 2637 (time 1788643171.33 total 30.02 last 83.13) - open/close 3489 (time 1788643181.34 total 40.02 last 85.16) - open/close 4309 (time 1788643191.34 total 50.02 last 81.97) - open/close 5247 (time 1788643201.34 total 60.03 last 93.78) - open/close 6119 (time 1788643211.35 total 70.03 last 87.15) - open/close 6969 (time 1788643221.36 total 80.04 last 84.92) - open/close 7923 (time 1788643231.36 total 90.05 last 95.35) - open/close 8897 (time 1788643241.37 total 100.05 last 97.32) - open/close 9915 (time 1788643251.37 total 110.06 last 101.79) total: 10000 open/close in 110.84 seconds: 90.22 ops/second - unlinked 0 (time 1788643256 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 982 (time 1788643330.09 total 10.00 last 98.17) - open/close 1860 (time 1788643340.11 total 20.02 last 87.63) - open/close 2810 (time 1788643350.11 total 30.03 last 94.96) - open/close 3696 (time 1788643360.11 total 40.03 last 88.59) - open/close 4565 (time 1788643370.12 total 50.03 last 86.86) - open/close 5473 (time 1788643380.13 total 60.04 last 90.72) - open/close 6537 (time 1788643390.13 total 70.04 last 106.39) - open/close 7618 (time 1788643400.15 total 80.06 last 107.89) - open/close 8587 (time 1788643410.16 total 90.08 last 96.77) - open/close 9552 (time 1788643420.17 total 100.09 last 96.43) total: 10000 open/close in 103.96 seconds: 96.19 ops/second - unlinked 0 (time 1788643428 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 987 (time 1788643500.06 total 10.00 last 98.66) - open/close 1965 (time 1788643510.08 total 20.02 last 97.65) - open/close 2906 (time 1788643520.08 total 30.02 last 94.08) - open/close 3921 (time 1788643530.08 total 40.02 last 101.49) - open/close 4895 (time 1788643540.10 total 50.04 last 97.23) - open/close 5907 (time 1788643550.11 total 60.05 last 101.05) - open/close 6801 (time 1788643560.12 total 70.06 last 89.37) - open/close 7708 (time 1788643570.12 total 80.06 last 90.66) - open/close 8684 (time 1788643580.12 total 90.06 last 97.60) - open/close 9654 (time 1788643590.13 total 100.07 last 96.94) total: 10000 open/close in 103.83 seconds: 96.31 ops/second - unlinked 0 (time 1788643597 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 978 (time 1788643670.46 total 10.01 last 97.71) - open/close 1861 (time 1788643680.46 total 20.01 last 88.30) - open/close 2821 (time 1788643690.46 total 30.01 last 95.96) - open/close 3782 (time 1788643700.47 total 40.02 last 96.05) - open/close 4824 (time 1788643710.48 total 50.03 last 104.09) - open/close 5810 (time 1788643720.48 total 60.04 last 98.54) - open/close 6777 (time 1788643730.49 total 70.04 last 96.67) - open/close 7825 (time 1788643740.49 total 80.04 last 104.78) - open/close 8730 (time 1788643750.49 total 90.04 last 90.48) - open/close 9514 (time 1788643760.51 total 100.06 last 78.28) total: 10000 open/close in 106.91 seconds: 93.54 ops/second - unlinked 0 (time 1788643771 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 845 (time 1788643849.60 total 10.01 last 84.44) - open/close 1588 (time 1788643859.61 total 20.01 last 74.27) - open/close 2536 (time 1788643869.61 total 30.01 last 94.77) - open/close 3466 (time 1788643879.62 total 40.02 last 92.96) - open/close 4352 (time 1788643889.62 total 50.03 last 88.54) - open/close 5264 (time 1788643899.63 total 60.03 last 91.16) - open/close 6069 (time 1788643909.63 total 70.04 last 80.44) - open/close 6975 (time 1788643919.64 total 80.04 last 90.59) - open/close 7946 (time 1788643929.64 total 90.04 last 97.06) - open/close 8885 (time 1788643939.66 total 100.06 last 93.75) - open/close 9630 (time 1788643949.66 total 110.06 last 74.46) total: 10000 open/close in 115.49 seconds: 86.59 ops/second - unlinked 0 (time 1788643959 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 961 (time 1788644033.20 total 10.01 last 96.01) - open/close 1939 (time 1788644043.21 total 20.01 last 97.79) - open/close 2837 (time 1788644053.21 total 30.01 last 89.77) - open/close 3851 (time 1788644063.21 total 40.02 last 101.36) - open/close 4768 (time 1788644073.22 total 50.02 last 91.66) - open/close 5719 (time 1788644083.22 total 60.03 last 95.03) - open/close 6706 (time 1788644093.23 total 70.03 last 98.65) - open/close 7623 (time 1788644103.24 total 80.05 last 91.60) - open/close 8483 (time 1788644113.25 total 90.05 last 85.95) - open/close 9377 (time 1788644123.25 total 100.05 last 89.37) total: 10000 open/close in 106.10 seconds: 94.25 ops/second - unlinked 0 (time 1788644133 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1027 (time 1788644209.39 total 10.00 last 102.69) - open/close 2012 (time 1788644219.40 total 20.01 last 98.38) - open/close 2997 (time 1788644229.43 total 30.05 last 98.19) - open/close 3949 (time 1788644239.43 total 40.05 last 95.18) - open/close 4875 (time 1788644249.44 total 50.05 last 92.56) - open/close 5687 (time 1788644259.46 total 60.07 last 81.05) - open/close 6546 (time 1788644269.48 total 70.09 last 85.69) - open/close 7169 (time 1788644279.49 total 80.11 last 62.23) - open/close 8078 (time 1788644289.50 total 90.11 last 90.85) - open/close 8906 (time 1788644299.50 total 100.11 last 82.78) - open/close 9757 (time 1788644309.50 total 110.12 last 85.09) total: 10000 open/close in 112.81 seconds: 88.65 ops/second - unlinked 0 (time 1788644315 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 823 (time 1788644394.51 total 10.03 last 82.02) - open/close 1708 (time 1788644404.52 total 20.04 last 88.45) - open/close 2456 (time 1788644414.52 total 30.04 last 74.79) - open/close 3280 (time 1788644424.52 total 40.05 last 82.36) - open/close 4169 (time 1788644434.53 total 50.05 last 88.86) - open/close 5086 (time 1788644444.54 total 60.07 last 91.55) - open/close 6000 (time 1788644454.55 total 70.07 last 91.34) - open/close 7003 (time 1788644464.55 total 80.07 last 100.28) - open/close 8003 (time 1788644474.56 total 90.08 last 99.96) - open/close 9039 (time 1788644484.56 total 100.08 last 103.55) total: 10000 open/close in 109.67 seconds: 91.18 ops/second - unlinked 0 (time 1788644497 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server stop mds service on oleg605-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg605-server stop mds service on oleg605-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg605-server unloading modules via unload_modules_local on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing unload_modules_local oleg605-server: kmemleak disabled modules unloaded. pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2