== conf-sanity test 69: replace an OST with the same index ========================================================== 12:32:43 (1788625963) start mds service on oleg315-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 17:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: kmemleak disabled oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg315-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 571 (time 1788626082.79 total 10.00 last 57.08) - open/close 1265 (time 1788626092.79 total 20.01 last 69.38) - open/close 2047 (time 1788626102.80 total 30.01 last 78.14) - open/close 2743 (time 1788626112.81 total 40.02 last 69.55) - open/close 3344 (time 1788626122.83 total 50.05 last 59.95) - open/close 3929 (time 1788626132.84 total 60.05 last 58.47) - open/close 4521 (time 1788626142.85 total 70.06 last 59.15) - open/close 5273 (time 1788626152.85 total 80.07 last 75.15) - open/close 6072 (time 1788626162.86 total 90.08 last 79.83) - open/close 6739 (time 1788626172.87 total 100.08 last 66.66) - open/close 7520 (time 1788626182.87 total 110.08 last 78.10) - open/close 8326 (time 1788626192.88 total 120.09 last 80.50) - open/close 9021 (time 1788626202.89 total 130.10 last 69.47) - open/close 9768 (time 1788626212.89 total 140.11 last 74.64) total: 10000 open/close in 143.54 seconds: 69.67 ops/second - unlinked 0 (time 1788626221 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 666 (time 1788626310.24 total 10.01 last 66.55) - open/close 1150 (time 1788626320.24 total 20.01 last 48.37) - open/close 1835 (time 1788626330.25 total 30.02 last 68.48) - open/close 2505 (time 1788626340.27 total 40.04 last 66.87) - open/close 3193 (time 1788626350.28 total 50.05 last 68.70) - open/close 3874 (time 1788626360.30 total 60.07 last 67.98) - open/close 4583 (time 1788626370.31 total 70.08 last 70.83) - open/close 5267 (time 1788626380.31 total 80.08 last 68.37) - open/close 5962 (time 1788626390.32 total 90.09 last 69.48) - open/close 6655 (time 1788626400.32 total 100.09 last 69.28) - open/close 7376 (time 1788626410.33 total 110.10 last 72.05) - open/close 8108 (time 1788626420.33 total 120.10 last 73.17) - open/close 8945 (time 1788626430.35 total 130.12 last 83.51) - open/close 9578 (time 1788626440.36 total 140.13 last 63.23) total: 10000 open/close in 145.84 seconds: 68.57 ops/second - unlinked 0 (time 1788626451 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 794 (time 1788626543.23 total 10.01 last 79.35) - open/close 1492 (time 1788626553.24 total 20.01 last 69.77) - open/close 2215 (time 1788626563.24 total 30.02 last 72.25) - open/close 2871 (time 1788626573.26 total 40.03 last 65.53) - open/close 3654 (time 1788626583.27 total 50.04 last 78.23) - open/close 4371 (time 1788626593.27 total 60.04 last 71.69) - open/close 5121 (time 1788626603.29 total 70.06 last 74.83) - open/close 5812 (time 1788626613.30 total 80.07 last 69.04) - open/close 6585 (time 1788626623.31 total 90.08 last 77.21) - open/close 7352 (time 1788626633.32 total 100.09 last 76.63) - open/close 8096 (time 1788626643.33 total 110.10 last 74.33) - open/close 8743 (time 1788626653.33 total 120.10 last 64.69) - open/close 9554 (time 1788626663.33 total 130.11 last 81.06) total: 10000 open/close in 135.99 seconds: 73.53 ops/second - unlinked 0 (time 1788626674 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 789 (time 1788626763.01 total 10.01 last 78.86) - open/close 1457 (time 1788626773.02 total 20.02 last 66.70) - open/close 2111 (time 1788626783.04 total 30.04 last 65.27) - open/close 2658 (time 1788626793.08 total 40.08 last 54.47) - open/close 3287 (time 1788626803.11 total 50.11 last 62.72) - open/close 3888 (time 1788626813.14 total 60.14 last 59.91) - open/close 4703 (time 1788626823.16 total 70.16 last 81.38) - open/close 5502 (time 1788626833.16 total 80.16 last 79.90) - open/close 6286 (time 1788626843.16 total 90.16 last 78.38) - open/close 7032 (time 1788626853.18 total 100.18 last 74.47) - open/close 7827 (time 1788626863.18 total 110.18 last 79.49) - open/close 8508 (time 1788626873.18 total 120.18 last 68.08) - open/close 9136 (time 1788626883.19 total 130.19 last 62.71) - open/close 9739 (time 1788626893.20 total 140.20 last 60.24) total: 10000 open/close in 144.44 seconds: 69.23 ops/second - unlinked 0 (time 1788626902 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 777 (time 1788626998.32 total 10.00 last 77.67) - open/close 1433 (time 1788627008.32 total 20.01 last 65.57) - open/close 2141 (time 1788627018.33 total 30.01 last 70.76) - open/close 2916 (time 1788627028.34 total 40.02 last 77.43) - open/close 3670 (time 1788627038.34 total 50.03 last 75.35) - open/close 4412 (time 1788627048.34 total 60.03 last 74.19) - open/close 5128 (time 1788627058.36 total 70.05 last 71.48) - open/close 5889 (time 1788627068.37 total 80.05 last 76.06) - open/close 6595 (time 1788627078.38 total 90.07 last 70.50) - open/close 7365 (time 1788627088.39 total 100.08 last 76.89) - open/close 8060 (time 1788627098.41 total 110.09 last 69.43) - open/close 8797 (time 1788627108.41 total 120.10 last 73.65) - open/close 9475 (time 1788627118.42 total 130.10 last 67.78) total: 10000 open/close in 138.26 seconds: 72.33 ops/second - unlinked 0 (time 1788627132 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 604 (time 1788627231.73 total 10.00 last 60.40) - open/close 1362 (time 1788627241.74 total 20.01 last 75.75) - open/close 2071 (time 1788627251.75 total 30.02 last 70.83) - open/close 2745 (time 1788627261.75 total 40.02 last 67.39) - open/close 3444 (time 1788627271.75 total 50.02 last 69.86) - open/close 4049 (time 1788627281.77 total 60.04 last 60.42) - open/close 4737 (time 1788627291.80 total 70.07 last 68.57) - open/close 5339 (time 1788627301.80 total 80.07 last 60.19) - open/close 5994 (time 1788627311.82 total 90.09 last 65.42) - open/close 6595 (time 1788627321.82 total 100.09 last 60.06) - open/close 7314 (time 1788627331.83 total 110.10 last 71.85) - open/close 7988 (time 1788627341.84 total 120.11 last 67.36) - open/close 8653 (time 1788627351.84 total 130.11 last 66.50) - open/close 9358 (time 1788627361.84 total 140.11 last 70.46) total: 10000 open/close in 149.08 seconds: 67.08 ops/second - unlinked 0 (time 1788627375 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 758 (time 1788627467.08 total 10.00 last 75.79) - open/close 1443 (time 1788627477.09 total 20.01 last 68.46) - open/close 2162 (time 1788627487.10 total 30.02 last 71.82) - open/close 2845 (time 1788627497.12 total 40.04 last 68.17) - open/close 3473 (time 1788627507.13 total 50.05 last 62.72) - open/close 4179 (time 1788627517.14 total 60.06 last 70.54) - open/close 4803 (time 1788627527.16 total 70.08 last 62.26) - open/close 5477 (time 1788627537.17 total 80.09 last 67.35) - open/close 6154 (time 1788627547.17 total 90.09 last 67.70) - open/close 6828 (time 1788627557.18 total 100.10 last 67.35) - open/close 7567 (time 1788627567.20 total 110.12 last 73.74) - open/close 8245 (time 1788627577.21 total 120.12 last 67.76) - open/close 8908 (time 1788627587.22 total 130.14 last 66.21) - open/close 9538 (time 1788627597.22 total 140.14 last 62.98) total: 10000 open/close in 146.19 seconds: 68.40 ops/second - unlinked 0 (time 1788627608 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 748 (time 1788627698.53 total 10.00 last 74.79) - open/close 1482 (time 1788627708.53 total 20.00 last 73.40) - open/close 2286 (time 1788627718.54 total 30.01 last 80.30) - open/close 2927 (time 1788627728.55 total 40.02 last 64.04) - open/close 3626 (time 1788627738.57 total 50.04 last 69.78) - open/close 4512 (time 1788627748.57 total 60.05 last 88.55) - open/close 5309 (time 1788627758.58 total 70.05 last 79.64) - open/close 6160 (time 1788627768.58 total 80.06 last 85.08) - open/close 6975 (time 1788627778.59 total 90.06 last 81.46) - open/close 7675 (time 1788627788.59 total 100.07 last 69.95) - open/close 8314 (time 1788627798.61 total 110.08 last 63.80) - open/close 8880 (time 1788627808.63 total 120.10 last 56.49) - open/close 9598 (time 1788627818.63 total 130.11 last 71.76) total: 10000 open/close in 135.79 seconds: 73.64 ops/second - unlinked 0 (time 1788627829 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 750 (time 1788627926.29 total 10.01 last 74.96) - open/close 1524 (time 1788627936.30 total 20.01 last 77.34) - open/close 2332 (time 1788627946.31 total 30.02 last 80.73) - open/close 3143 (time 1788627956.31 total 40.02 last 81.09) - open/close 3888 (time 1788627966.31 total 50.03 last 74.48) - open/close 4665 (time 1788627976.34 total 60.06 last 77.45) - open/close 5446 (time 1788627986.35 total 70.06 last 78.08) - open/close 6201 (time 1788627996.35 total 80.07 last 75.47) - open/close 6920 (time 1788628006.36 total 90.07 last 71.83) - open/close 7676 (time 1788628016.37 total 100.08 last 75.56) - open/close 8462 (time 1788628026.37 total 110.08 last 78.56) - open/close 9261 (time 1788628036.38 total 120.09 last 79.83) total: 10000 open/close in 130.09 seconds: 76.87 ops/second - unlinked 0 (time 1788628051 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 709 (time 1788628142.62 total 10.01 last 70.86) - open/close 1418 (time 1788628152.62 total 20.01 last 70.90) - open/close 2119 (time 1788628162.64 total 30.03 last 69.92) - open/close 2835 (time 1788628172.65 total 40.04 last 71.55) - open/close 3488 (time 1788628182.66 total 50.05 last 65.26) - open/close 4255 (time 1788628192.66 total 60.05 last 76.66) - open/close 4978 (time 1788628202.67 total 70.06 last 72.25) - open/close 5740 (time 1788628212.67 total 80.06 last 76.19) - open/close 6539 (time 1788628222.67 total 90.06 last 79.90) - open/close 7315 (time 1788628232.68 total 100.07 last 77.55) - open/close 8090 (time 1788628242.69 total 110.08 last 77.38) - open/close 8844 (time 1788628252.70 total 120.09 last 75.34) - open/close 9622 (time 1788628262.70 total 130.09 last 77.79) total: 10000 open/close in 134.79 seconds: 74.19 ops/second - unlinked 0 (time 1788628271 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: kmemleak disabled modules unloaded. pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2