== conf-sanity test 69: replace an OST with the same index ========================================================== 14:01:38 (1790964098) start mds service on oleg219-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Oct 2 13:44 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: kmemleak disabled oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: MODOPTS_LIBCFS= oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg219-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg219-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg219-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1226 (time 1790964199.67 total 10.00 last 122.59) - open/close 2406 (time 1790964209.68 total 20.01 last 117.87) - open/close 3528 (time 1790964219.69 total 30.02 last 112.10) - open/close 4822 (time 1790964229.69 total 40.03 last 129.34) - open/close 6101 (time 1790964239.70 total 50.03 last 127.84) - open/close 7446 (time 1790964249.70 total 60.03 last 134.47) - open/close 8347 (time 1790964259.72 total 70.05 last 89.95) - open/close 9140 (time 1790964269.72 total 80.05 last 79.29) total: 10000 open/close in 89.45 seconds: 111.79 ops/second - unlinked 0 (time 1790964283 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 905 (time 1790964353.45 total 10.00 last 90.48) - open/close 1926 (time 1790964363.46 total 20.01 last 102.01) - open/close 3023 (time 1790964373.46 total 30.01 last 109.69) - open/close 3881 (time 1790964383.46 total 40.01 last 85.78) - open/close 4937 (time 1790964393.47 total 50.02 last 105.52) - open/close 5955 (time 1790964403.47 total 60.03 last 101.73) - open/close 7176 (time 1790964413.47 total 70.03 last 122.08) - open/close 8133 (time 1790964423.48 total 80.03 last 95.68) - open/close 9034 (time 1790964433.51 total 90.06 last 89.82) - open/close 9885 (time 1790964443.51 total 100.07 last 85.07) total: 10000 open/close in 101.50 seconds: 98.52 ops/second - unlinked 0 (time 1790964448 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1083 (time 1790964516.27 total 10.00 last 108.30) - open/close 2071 (time 1790964526.28 total 20.01 last 98.75) - open/close 2999 (time 1790964536.28 total 30.01 last 92.78) - open/close 3983 (time 1790964546.28 total 40.01 last 98.39) - open/close 4938 (time 1790964556.30 total 50.03 last 95.29) - open/close 5886 (time 1790964566.31 total 60.04 last 94.72) - open/close 6686 (time 1790964576.32 total 70.05 last 79.95) - open/close 7461 (time 1790964586.32 total 80.05 last 77.49) - open/close 8317 (time 1790964596.34 total 90.07 last 85.40) - open/close 9165 (time 1790964606.35 total 100.08 last 84.76) total: 10000 open/close in 108.85 seconds: 91.87 ops/second - unlinked 0 (time 1790964618 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 821 (time 1790964699.48 total 10.03 last 81.88) - open/close 1482 (time 1790964709.50 total 20.05 last 65.94) - open/close 2304 (time 1790964719.51 total 30.06 last 82.11) - open/close 2935 (time 1790964729.63 total 40.17 last 62.39) - open/close 3592 (time 1790964739.66 total 50.20 last 65.51) - open/close 4248 (time 1790964749.66 total 60.20 last 65.59) - open/close 5278 (time 1790964759.66 total 70.21 last 102.98) - open/close 6024 (time 1790964769.66 total 80.21 last 74.56) - open/close 6715 (time 1790964779.67 total 90.21 last 69.09) - open/close 7575 (time 1790964789.67 total 100.22 last 85.95) - open/close 8370 (time 1790964799.68 total 110.23 last 79.39) - open/close 9125 (time 1790964809.69 total 120.24 last 75.47) - open/close 9916 (time 1790964819.69 total 130.24 last 79.09) total: 10000 open/close in 131.43 seconds: 76.09 ops/second - unlinked 0 (time 1790964825 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 863 (time 1790964910.44 total 10.01 last 86.20) - open/close 1722 (time 1790964920.45 total 20.02 last 85.84) - open/close 2515 (time 1790964930.46 total 30.04 last 79.16) - open/close 3346 (time 1790964940.47 total 40.05 last 83.00) - open/close 4174 (time 1790964950.48 total 50.05 last 82.75) - open/close 5021 (time 1790964960.49 total 60.06 last 84.63) - open/close 5775 (time 1790964970.50 total 70.07 last 75.34) - open/close 6527 (time 1790964980.50 total 80.07 last 75.19) - open/close 7411 (time 1790964990.50 total 90.07 last 88.40) - open/close 8351 (time 1790965000.50 total 100.08 last 93.95) - open/close 9230 (time 1790965010.51 total 110.08 last 87.82) total: 10000 open/close in 118.66 seconds: 84.28 ops/second - unlinked 0 (time 1790965023 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 941 (time 1790965112.18 total 10.00 last 94.07) - open/close 1797 (time 1790965122.18 total 20.00 last 85.60) - open/close 2717 (time 1790965132.18 total 30.00 last 91.99) - open/close 3611 (time 1790965142.19 total 40.01 last 89.32) - open/close 4422 (time 1790965152.21 total 50.03 last 80.97) - open/close 5381 (time 1790965162.21 total 60.03 last 95.86) - open/close 6211 (time 1790965172.21 total 70.03 last 82.99) - open/close 7053 (time 1790965182.22 total 80.04 last 84.16) - open/close 7987 (time 1790965192.22 total 90.04 last 93.37) - open/close 8858 (time 1790965202.22 total 100.05 last 87.05) - open/close 9819 (time 1790965212.23 total 110.05 last 96.06) total: 10000 open/close in 112.64 seconds: 88.77 ops/second - unlinked 0 (time 1790965219 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 864 (time 1790965308.87 total 10.00 last 86.39) - open/close 1771 (time 1790965318.88 total 20.01 last 90.62) - open/close 2546 (time 1790965328.89 total 30.02 last 77.43) - open/close 3445 (time 1790965338.89 total 40.03 last 89.83) - open/close 4281 (time 1790965348.90 total 50.04 last 83.52) - open/close 5092 (time 1790965358.91 total 60.05 last 81.03) - open/close 5928 (time 1790965368.92 total 70.06 last 83.51) - open/close 6750 (time 1790965378.94 total 80.07 last 82.09) - open/close 7642 (time 1790965388.94 total 90.07 last 89.18) - open/close 8637 (time 1790965398.95 total 100.08 last 99.42) - open/close 9523 (time 1790965408.96 total 110.09 last 88.53) total: 10000 open/close in 115.92 seconds: 86.27 ops/second - unlinked 0 (time 1790965419 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 871 (time 1790965501.72 total 10.02 last 86.95) - open/close 1687 (time 1790965511.75 total 20.04 last 81.39) - open/close 2614 (time 1790965521.76 total 30.05 last 92.63) - open/close 3449 (time 1790965531.77 total 40.06 last 83.41) - open/close 4308 (time 1790965541.77 total 50.06 last 85.87) - open/close 4942 (time 1790965551.79 total 60.08 last 63.32) - open/close 5386 (time 1790965561.80 total 70.09 last 44.36) - open/close 5902 (time 1790965571.80 total 80.09 last 51.58) - open/close 6780 (time 1790965581.80 total 90.09 last 87.78) - open/close 7412 (time 1790965591.82 total 100.11 last 63.10) - open/close 8388 (time 1790965601.82 total 110.11 last 97.59) - open/close 9253 (time 1790965611.82 total 120.11 last 86.47) total: 10000 open/close in 127.12 seconds: 78.67 ops/second - unlinked 0 (time 1790965623 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1022 (time 1790965705.04 total 10.00 last 102.15) - open/close 1742 (time 1790965715.05 total 20.02 last 71.90) - open/close 2508 (time 1790965725.06 total 30.02 last 76.55) - open/close 3252 (time 1790965735.06 total 40.03 last 74.37) - open/close 3982 (time 1790965745.07 total 50.03 last 72.97) - open/close 4859 (time 1790965755.07 total 60.04 last 87.66) - open/close 5667 (time 1790965765.08 total 70.04 last 80.75) - open/close 6459 (time 1790965775.08 total 80.05 last 79.17) - open/close 7244 (time 1790965785.09 total 90.05 last 78.47) - open/close 8144 (time 1790965795.11 total 100.07 last 89.80) - open/close 9094 (time 1790965805.11 total 110.08 last 94.96) - open/close 9981 (time 1790965815.12 total 120.09 last 88.63) total: 10000 open/close in 120.34 seconds: 83.10 ops/second - unlinked 0 (time 1790965819 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 828 (time 1790965904.26 total 10.00 last 82.77) - open/close 1592 (time 1790965914.28 total 20.02 last 76.27) - open/close 2460 (time 1790965924.28 total 30.02 last 86.79) - open/close 3403 (time 1790965934.28 total 40.02 last 94.30) - open/close 4196 (time 1790965944.28 total 50.02 last 79.29) - open/close 4984 (time 1790965954.30 total 60.04 last 78.66) - open/close 5807 (time 1790965964.30 total 70.05 last 82.23) - open/close 6570 (time 1790965974.31 total 80.06 last 76.22) - open/close 7366 (time 1790965984.32 total 90.06 last 79.57) - open/close 8307 (time 1790965994.33 total 100.07 last 94.02) - open/close 9009 (time 1790966004.33 total 110.08 last 70.16) - open/close 9644 (time 1790966014.33 total 120.08 last 63.49) total: 10000 open/close in 125.84 seconds: 79.46 ops/second - unlinked 0 (time 1790966026 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg219-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server unloading modules via unload_modules_local on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: kmemleak disabled modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2