== conf-sanity test 69: replace an OST with the same index ========================================================== 08:12:44 (1788437564) start mds service on oleg121-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: kmemleak disabled oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg121-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg121-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1131 (time 1788437675.97 total 10.01 last 113.01) - open/close 2128 (time 1788437685.98 total 20.01 last 99.64) - open/close 3124 (time 1788437695.98 total 30.02 last 99.54) - open/close 4291 (time 1788437705.99 total 40.03 last 116.62) - open/close 5333 (time 1788437715.99 total 50.03 last 104.16) - open/close 6348 (time 1788437726.00 total 60.03 last 101.47) - open/close 7261 (time 1788437736.00 total 70.04 last 91.23) - open/close 8238 (time 1788437746.01 total 80.05 last 97.63) - open/close 9161 (time 1788437756.02 total 90.06 last 92.21) total: 10000 open/close in 99.08 seconds: 100.93 ops/second - unlinked 0 (time 1788437770 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 930 (time 1788437848.58 total 10.00 last 92.96) - open/close 1966 (time 1788437858.58 total 20.01 last 103.54) - open/close 3135 (time 1788437868.59 total 30.02 last 116.84) - open/close 4179 (time 1788437878.59 total 40.02 last 104.37) - open/close 5027 (time 1788437888.59 total 50.02 last 84.76) - open/close 6100 (time 1788437898.60 total 60.03 last 107.27) - open/close 7085 (time 1788437908.61 total 70.04 last 98.36) - open/close 8154 (time 1788437918.62 total 80.05 last 106.84) - open/close 9144 (time 1788437928.62 total 90.05 last 98.95) total: 10000 open/close in 98.84 seconds: 101.17 ops/second - unlinked 0 (time 1788437940 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1119 (time 1788438009.60 total 10.02 last 111.73) - open/close 2117 (time 1788438019.61 total 20.02 last 99.76) - open/close 3209 (time 1788438029.61 total 30.03 last 109.12) - open/close 4243 (time 1788438039.61 total 40.03 last 103.40) - open/close 5268 (time 1788438049.62 total 50.03 last 102.45) - open/close 6279 (time 1788438059.62 total 60.03 last 101.07) - open/close 7325 (time 1788438069.62 total 70.04 last 104.60) - open/close 8408 (time 1788438079.62 total 80.04 last 108.27) - open/close 9512 (time 1788438089.63 total 90.04 last 110.34) total: 10000 open/close in 94.62 seconds: 105.69 ops/second - unlinked 0 (time 1788438097 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1022 (time 1788438165.86 total 10.01 last 102.13) - open/close 1906 (time 1788438175.87 total 20.02 last 88.28) - open/close 2854 (time 1788438185.88 total 30.03 last 94.75) - open/close 3860 (time 1788438195.88 total 40.03 last 100.58) - open/close 4990 (time 1788438205.88 total 50.03 last 112.97) - open/close 6029 (time 1788438215.89 total 60.04 last 103.76) - open/close 6999 (time 1788438225.90 total 70.05 last 96.90) - open/close 7897 (time 1788438235.91 total 80.06 last 89.76) - open/close 8976 (time 1788438245.91 total 90.06 last 107.85) - open/close 9924 (time 1788438255.91 total 100.06 last 94.77) total: 10000 open/close in 100.82 seconds: 99.19 ops/second - unlinked 0 (time 1788438260 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1072 (time 1788438333.61 total 10.01 last 107.13) - open/close 1991 (time 1788438343.61 total 20.01 last 91.88) - open/close 3046 (time 1788438353.62 total 30.01 last 105.45) - open/close 4188 (time 1788438363.63 total 40.02 last 114.09) - open/close 5233 (time 1788438373.64 total 50.04 last 104.38) - open/close 6330 (time 1788438383.64 total 60.04 last 109.68) - open/close 7156 (time 1788438393.64 total 70.04 last 82.56) - open/close 8164 (time 1788438403.65 total 80.05 last 100.75) - open/close 9038 (time 1788438413.66 total 90.06 last 87.27) total: 10000 open/close in 99.94 seconds: 100.06 ops/second - unlinked 0 (time 1788438428 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 968 (time 1788438498.06 total 10.01 last 96.67) - open/close 2037 (time 1788438508.06 total 20.02 last 106.83) - open/close 3020 (time 1788438518.07 total 30.03 last 98.21) - open/close 3928 (time 1788438528.09 total 40.05 last 90.62) - open/close 4889 (time 1788438538.10 total 50.06 last 96.01) - open/close 5903 (time 1788438548.10 total 60.06 last 101.37) - open/close 6883 (time 1788438558.10 total 70.06 last 98.00) - open/close 7821 (time 1788438568.11 total 80.06 last 93.77) - open/close 8856 (time 1788438578.12 total 90.07 last 103.41) - open/close 9730 (time 1788438588.12 total 100.08 last 87.38) total: 10000 open/close in 103.10 seconds: 96.99 ops/second - unlinked 0 (time 1788438596 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1066 (time 1788438671.77 total 10.01 last 106.48) - open/close 2044 (time 1788438681.77 total 20.01 last 97.78) - open/close 2996 (time 1788438691.80 total 30.04 last 94.91) - open/close 3660 (time 1788438701.81 total 40.05 last 66.39) - open/close 4136 (time 1788438711.81 total 50.05 last 47.57) - open/close 4736 (time 1788438721.82 total 60.06 last 59.96) - open/close 5767 (time 1788438731.83 total 70.07 last 102.93) - open/close 6628 (time 1788438741.84 total 80.08 last 86.05) - open/close 7524 (time 1788438751.85 total 90.09 last 89.52) - open/close 8451 (time 1788438761.85 total 100.09 last 92.69) - open/close 9457 (time 1788438771.86 total 110.10 last 100.52) total: 10000 open/close in 116.00 seconds: 86.21 ops/second - unlinked 0 (time 1788438782 ; total 1 ; last 1) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 883 (time 1788438862.78 total 10.01 last 88.24) - open/close 1773 (time 1788438872.78 total 20.01 last 88.97) - open/close 2809 (time 1788438882.79 total 30.01 last 103.56) - open/close 3679 (time 1788438892.79 total 40.02 last 86.97) - open/close 4520 (time 1788438902.80 total 50.03 last 84.02) - open/close 5216 (time 1788438912.82 total 60.04 last 69.48) - open/close 5847 (time 1788438922.82 total 70.05 last 63.07) - open/close 6885 (time 1788438932.83 total 80.05 last 103.74) - open/close 7766 (time 1788438942.83 total 90.06 last 88.08) - open/close 8677 (time 1788438952.83 total 100.06 last 91.08) - open/close 9419 (time 1788438962.83 total 110.06 last 74.20) total: 10000 open/close in 116.85 seconds: 85.58 ops/second - unlinked 0 (time 1788438973 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 911 (time 1788439044.45 total 10.00 last 91.09) - open/close 1690 (time 1788439054.46 total 20.00 last 77.87) - open/close 2429 (time 1788439064.46 total 30.01 last 73.86) - open/close 3259 (time 1788439074.49 total 40.03 last 82.82) - open/close 4113 (time 1788439084.51 total 50.05 last 85.23) - open/close 5083 (time 1788439094.51 total 60.06 last 96.93) - open/close 6141 (time 1788439104.52 total 70.07 last 105.73) - open/close 7102 (time 1788439114.53 total 80.07 last 96.03) - open/close 8107 (time 1788439124.54 total 90.09 last 100.34) - open/close 9050 (time 1788439134.55 total 100.10 last 94.21) - open/close 9952 (time 1788439144.56 total 110.11 last 90.14) total: 10000 open/close in 110.59 seconds: 90.43 ops/second - unlinked 0 (time 1788439148 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1031 (time 1788439220.65 total 10.02 last 102.93) - open/close 1985 (time 1788439230.66 total 20.03 last 95.29) - open/close 3038 (time 1788439240.68 total 30.05 last 105.08) - open/close 4040 (time 1788439250.69 total 40.05 last 100.15) - open/close 4718 (time 1788439260.70 total 50.06 last 67.73) - open/close 5776 (time 1788439270.70 total 60.07 last 105.75) - open/close 6653 (time 1788439280.72 total 70.08 last 87.58) - open/close 7340 (time 1788439290.72 total 80.09 last 68.67) - open/close 8060 (time 1788439300.74 total 90.11 last 71.86) - open/close 8910 (time 1788439310.75 total 100.12 last 84.93) - open/close 9641 (time 1788439320.75 total 110.12 last 73.07) total: 10000 open/close in 112.80 seconds: 88.65 ops/second - unlinked 0 (time 1788439327 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg121-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.121@tcp sys.timeout=20 start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg121-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local oleg121-server: kmemleak disabled modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2