== conf-sanity test 69: replace an OST with the same index ========================================================== 15:31:32 (1788636692) start mds service on oleg153-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: kmemleak disabled oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg153-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg153-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg153-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 884 (time 1788636808.24 total 10.00 last 88.37) - open/close 1630 (time 1788636818.25 total 20.01 last 74.55) - open/close 2570 (time 1788636828.26 total 30.02 last 93.92) - open/close 3557 (time 1788636838.27 total 40.03 last 98.63) - open/close 4478 (time 1788636848.28 total 50.04 last 91.93) - open/close 5597 (time 1788636858.29 total 60.05 last 111.79) - open/close 6263 (time 1788636868.31 total 70.07 last 66.50) - open/close 6970 (time 1788636878.32 total 80.08 last 70.66) - open/close 7596 (time 1788636888.33 total 90.09 last 62.54) - open/close 8151 (time 1788636898.33 total 100.09 last 55.48) - open/close 9143 (time 1788636908.33 total 110.09 last 99.20) total: 10000 open/close in 119.50 seconds: 83.68 ops/second - unlinked 0 (time 1788636922 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 930 (time 1788636993.29 total 10.00 last 92.98) - open/close 1684 (time 1788637003.30 total 20.01 last 75.35) - open/close 2696 (time 1788637013.30 total 30.02 last 101.12) - open/close 3654 (time 1788637023.31 total 40.02 last 95.79) - open/close 4651 (time 1788637033.31 total 50.02 last 99.64) - open/close 5637 (time 1788637043.32 total 60.03 last 98.55) - open/close 6605 (time 1788637053.32 total 70.03 last 96.79) - open/close 7654 (time 1788637063.32 total 80.03 last 104.88) - open/close 8504 (time 1788637073.32 total 90.03 last 85.00) - open/close 9453 (time 1788637083.32 total 100.03 last 94.90) total: 10000 open/close in 105.03 seconds: 95.21 ops/second - unlinked 0 (time 1788637092 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1135 (time 1788637163.40 total 10.01 last 113.43) - open/close 2163 (time 1788637173.40 total 20.01 last 102.76) - open/close 3260 (time 1788637183.40 total 30.01 last 109.68) - open/close 4257 (time 1788637193.40 total 40.01 last 99.67) - open/close 5324 (time 1788637203.41 total 50.02 last 106.68) - open/close 6299 (time 1788637213.41 total 60.02 last 97.47) - open/close 7280 (time 1788637223.42 total 70.03 last 98.01) - open/close 8366 (time 1788637233.43 total 80.04 last 108.51) - open/close 9196 (time 1788637243.43 total 90.04 last 82.95) total: 10000 open/close in 98.02 seconds: 102.02 ops/second - unlinked 0 (time 1788637254 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 957 (time 1788637326.44 total 10.00 last 95.69) - open/close 1794 (time 1788637336.44 total 20.01 last 83.65) - open/close 2820 (time 1788637346.45 total 30.01 last 102.53) - open/close 3816 (time 1788637356.46 total 40.02 last 99.53) - open/close 4757 (time 1788637366.47 total 50.03 last 94.01) - open/close 5603 (time 1788637376.48 total 60.04 last 84.54) - open/close 6357 (time 1788637386.49 total 70.06 last 75.27) - open/close 7201 (time 1788637396.49 total 80.06 last 84.39) - open/close 7905 (time 1788637406.50 total 90.06 last 70.39) - open/close 8852 (time 1788637416.51 total 100.08 last 94.53) - open/close 9747 (time 1788637426.52 total 110.08 last 89.49) total: 10000 open/close in 113.18 seconds: 88.35 ops/second - unlinked 0 (time 1788637433 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1008 (time 1788637504.90 total 10.01 last 100.71) - open/close 2032 (time 1788637514.91 total 20.01 last 102.35) - open/close 3092 (time 1788637524.91 total 30.01 last 105.99) - open/close 4226 (time 1788637534.91 total 40.02 last 113.36) - open/close 5280 (time 1788637544.91 total 50.02 last 105.40) - open/close 6184 (time 1788637554.91 total 60.02 last 90.37) - open/close 6876 (time 1788637564.93 total 70.04 last 69.10) - open/close 7507 (time 1788637574.95 total 80.05 last 62.99) - open/close 8217 (time 1788637584.95 total 90.06 last 70.97) - open/close 8992 (time 1788637594.96 total 100.07 last 77.41) - open/close 9757 (time 1788637604.97 total 110.07 last 76.47) total: 10000 open/close in 112.64 seconds: 88.78 ops/second - unlinked 0 (time 1788637612 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1020 (time 1788637681.59 total 10.00 last 101.98) - open/close 1934 (time 1788637691.60 total 20.01 last 91.31) - open/close 2849 (time 1788637701.62 total 30.03 last 91.31) - open/close 3847 (time 1788637711.62 total 40.04 last 99.75) - open/close 4785 (time 1788637721.63 total 50.04 last 93.76) - open/close 5755 (time 1788637731.63 total 60.04 last 96.97) - open/close 6835 (time 1788637741.64 total 70.05 last 107.92) - open/close 7896 (time 1788637751.66 total 80.07 last 105.87) - open/close 9029 (time 1788637761.67 total 90.08 last 113.20) total: 10000 open/close in 98.58 seconds: 101.44 ops/second - unlinked 0 (time 1788637773 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 962 (time 1788637843.45 total 10.00 last 96.18) - open/close 1999 (time 1788637853.46 total 20.01 last 103.60) - open/close 2967 (time 1788637863.49 total 30.04 last 96.51) - open/close 4102 (time 1788637873.49 total 40.05 last 113.44) - open/close 5198 (time 1788637883.50 total 50.05 last 109.56) - open/close 6209 (time 1788637893.50 total 60.05 last 101.06) - open/close 7056 (time 1788637903.52 total 70.07 last 84.57) - open/close 7888 (time 1788637913.52 total 80.07 last 83.17) - open/close 8972 (time 1788637923.53 total 90.09 last 108.27) - open/close 9931 (time 1788637933.55 total 100.10 last 95.77) total: 10000 open/close in 100.78 seconds: 99.22 ops/second - unlinked 0 (time 1788637938 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 884 (time 1788638006.65 total 10.00 last 88.36) - open/close 1896 (time 1788638016.65 total 20.01 last 101.15) - open/close 3068 (time 1788638026.66 total 30.02 last 117.11) - open/close 4214 (time 1788638036.68 total 40.03 last 114.42) - open/close 5121 (time 1788638046.68 total 50.04 last 90.67) - open/close 6160 (time 1788638056.68 total 60.04 last 103.87) - open/close 7131 (time 1788638066.68 total 70.04 last 97.09) - open/close 8158 (time 1788638076.68 total 80.04 last 102.69) - open/close 9147 (time 1788638086.69 total 90.04 last 98.86) total: 10000 open/close in 97.82 seconds: 102.23 ops/second - unlinked 0 (time 1788638098 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1009 (time 1788638173.11 total 10.01 last 100.85) - open/close 1984 (time 1788638183.12 total 20.01 last 97.46) - open/close 2942 (time 1788638193.12 total 30.01 last 95.78) - open/close 3960 (time 1788638203.12 total 40.01 last 101.80) - open/close 4925 (time 1788638213.12 total 50.01 last 96.48) - open/close 5574 (time 1788638223.13 total 60.02 last 64.83) - open/close 6458 (time 1788638233.14 total 70.03 last 88.32) - open/close 7194 (time 1788638243.14 total 80.04 last 73.57) - open/close 7865 (time 1788638253.15 total 90.04 last 67.06) - open/close 8699 (time 1788638263.16 total 100.05 last 83.32) - open/close 9759 (time 1788638273.16 total 110.06 last 105.96) total: 10000 open/close in 112.01 seconds: 89.28 ops/second - unlinked 0 (time 1788638279 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1118 (time 1788638349.27 total 10.00 last 111.74) - open/close 2067 (time 1788638359.30 total 20.04 last 94.59) - open/close 2919 (time 1788638369.31 total 30.04 last 85.14) - open/close 3690 (time 1788638379.31 total 40.05 last 77.08) - open/close 4725 (time 1788638389.31 total 50.05 last 103.46) - open/close 5722 (time 1788638399.33 total 60.07 last 99.55) - open/close 6892 (time 1788638409.33 total 70.07 last 116.99) - open/close 7750 (time 1788638419.34 total 80.07 last 85.75) - open/close 8593 (time 1788638429.35 total 90.08 last 84.20) - open/close 9578 (time 1788638439.35 total 100.09 last 98.44) total: 10000 open/close in 104.54 seconds: 95.66 ops/second - unlinked 0 (time 1788638448 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local oleg153-server: kmemleak disabled modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2