== conf-sanity test 69: replace an OST with the same index ========================================================== 14:14:34 (1790100874) start mds service on oleg323-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 22 12:10 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: kmemleak disabled oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg323-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg323-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 846 (time 1790100988.83 total 10.01 last 84.55) - open/close 1693 (time 1790100998.84 total 20.01 last 84.66) - open/close 2527 (time 1790101008.85 total 30.02 last 83.30) - open/close 3362 (time 1790101018.86 total 40.04 last 83.40) - open/close 4225 (time 1790101028.87 total 50.04 last 86.24) - open/close 5144 (time 1790101038.87 total 60.04 last 91.89) - open/close 5981 (time 1790101048.88 total 70.05 last 83.63) - open/close 6706 (time 1790101058.89 total 80.06 last 72.43) - open/close 7428 (time 1790101068.90 total 90.07 last 72.15) - open/close 8212 (time 1790101078.90 total 100.07 last 78.39) - open/close 9087 (time 1790101088.92 total 110.09 last 87.30) - open/close 9925 (time 1790101098.93 total 120.10 last 83.72) total: 10000 open/close in 120.98 seconds: 82.65 ops/second - unlinked 0 (time 1790101104 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 781 (time 1790101183.99 total 10.00 last 78.08) - open/close 1542 (time 1790101194.00 total 20.01 last 76.06) - open/close 2486 (time 1790101204.02 total 30.03 last 94.22) - open/close 3399 (time 1790101214.03 total 40.04 last 91.16) - open/close 4323 (time 1790101224.04 total 50.05 last 92.34) - open/close 5129 (time 1790101234.04 total 60.06 last 80.55) - open/close 5908 (time 1790101244.05 total 70.06 last 77.87) - open/close 6799 (time 1790101254.06 total 80.07 last 89.01) - open/close 7690 (time 1790101264.06 total 90.08 last 89.04) - open/close 8641 (time 1790101274.07 total 100.08 last 95.05) - open/close 9553 (time 1790101284.07 total 110.08 last 91.19) total: 10000 open/close in 115.38 seconds: 86.67 ops/second - unlinked 0 (time 1790101293 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 868 (time 1790101370.79 total 10.01 last 86.74) - open/close 1794 (time 1790101380.80 total 20.01 last 92.54) - open/close 2767 (time 1790101390.80 total 30.02 last 97.24) - open/close 3651 (time 1790101400.81 total 40.03 last 88.34) - open/close 4605 (time 1790101410.82 total 50.03 last 95.34) - open/close 5525 (time 1790101420.82 total 60.04 last 91.95) - open/close 6513 (time 1790101430.83 total 70.04 last 98.73) - open/close 7334 (time 1790101440.84 total 80.05 last 82.04) - open/close 8267 (time 1790101450.84 total 90.05 last 93.28) - open/close 9056 (time 1790101460.84 total 100.06 last 78.89) - open/close 9983 (time 1790101470.85 total 110.06 last 92.64) total: 10000 open/close in 110.22 seconds: 90.72 ops/second - unlinked 0 (time 1790101474 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 898 (time 1790101550.42 total 10.01 last 89.75) - open/close 1833 (time 1790101560.42 total 20.01 last 93.49) - open/close 2799 (time 1790101570.43 total 30.01 last 96.53) - open/close 3713 (time 1790101580.44 total 40.02 last 91.36) - open/close 4702 (time 1790101590.46 total 50.04 last 98.69) - open/close 5730 (time 1790101600.48 total 60.06 last 102.60) - open/close 6586 (time 1790101610.48 total 70.07 last 85.55) - open/close 7582 (time 1790101620.49 total 80.08 last 99.50) - open/close 8492 (time 1790101630.51 total 90.09 last 90.88) - open/close 9376 (time 1790101640.51 total 100.09 last 88.40) total: 10000 open/close in 107.03 seconds: 93.44 ops/second - unlinked 0 (time 1790101650 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 887 (time 1790101726.31 total 10.01 last 88.66) - open/close 1706 (time 1790101736.31 total 20.01 last 81.85) - open/close 2450 (time 1790101746.31 total 30.01 last 74.38) - open/close 3057 (time 1790101756.32 total 40.02 last 60.64) - open/close 3977 (time 1790101766.32 total 50.02 last 91.99) - open/close 4972 (time 1790101776.34 total 60.04 last 99.38) - open/close 5840 (time 1790101786.34 total 70.04 last 86.77) - open/close 6687 (time 1790101796.35 total 80.05 last 84.65) - open/close 7647 (time 1790101806.35 total 90.05 last 95.97) - open/close 8486 (time 1790101816.36 total 100.06 last 83.82) - open/close 9374 (time 1790101826.36 total 110.06 last 88.78) total: 10000 open/close in 117.23 seconds: 85.31 ops/second - unlinked 0 (time 1790101837 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 916 (time 1790101913.76 total 10.01 last 91.52) - open/close 1766 (time 1790101923.79 total 20.04 last 84.76) - open/close 2690 (time 1790101933.80 total 30.05 last 92.26) - open/close 3610 (time 1790101943.81 total 40.06 last 91.91) - open/close 4488 (time 1790101953.82 total 50.07 last 87.76) - open/close 5416 (time 1790101963.82 total 60.07 last 92.80) - open/close 6239 (time 1790101973.82 total 70.07 last 82.27) - open/close 7160 (time 1790101983.83 total 80.08 last 92.01) - open/close 8115 (time 1790101993.83 total 90.08 last 95.48) - open/close 9015 (time 1790102003.84 total 100.09 last 89.91) - open/close 9968 (time 1790102013.85 total 110.10 last 95.27) total: 10000 open/close in 110.38 seconds: 90.60 ops/second - unlinked 0 (time 1790102018 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 949 (time 1790102098.19 total 10.00 last 94.90) - open/close 1866 (time 1790102108.19 total 20.00 last 91.67) - open/close 2819 (time 1790102118.20 total 30.01 last 95.24) - open/close 3736 (time 1790102128.20 total 40.01 last 91.68) - open/close 4631 (time 1790102138.20 total 50.01 last 89.49) - open/close 5470 (time 1790102148.20 total 60.02 last 83.88) - open/close 6367 (time 1790102158.20 total 70.02 last 89.68) - open/close 7302 (time 1790102168.21 total 80.02 last 93.44) - open/close 8256 (time 1790102178.21 total 90.03 last 95.39) - open/close 9161 (time 1790102188.23 total 100.04 last 90.33) total: 10000 open/close in 109.34 seconds: 91.46 ops/second - unlinked 0 (time 1790102201 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 849 (time 1790102278.45 total 10.00 last 84.86) - open/close 1737 (time 1790102288.45 total 20.01 last 88.79) - open/close 2505 (time 1790102298.46 total 30.02 last 76.68) - open/close 3361 (time 1790102308.47 total 40.03 last 85.54) - open/close 4164 (time 1790102318.48 total 50.03 last 80.25) - open/close 4715 (time 1790102328.50 total 60.06 last 54.98) - open/close 5565 (time 1790102338.50 total 70.06 last 84.97) - open/close 6505 (time 1790102348.50 total 80.06 last 93.98) - open/close 7353 (time 1790102358.50 total 90.06 last 84.79) - open/close 8243 (time 1790102368.51 total 100.07 last 88.92) - open/close 9093 (time 1790102378.52 total 110.08 last 84.98) - open/close 9933 (time 1790102388.53 total 120.09 last 83.91) total: 10000 open/close in 121.11 seconds: 82.57 ops/second - unlinked 0 (time 1790102393 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 943 (time 1790102469.97 total 10.02 last 94.10) - open/close 1775 (time 1790102479.98 total 20.03 last 83.16) - open/close 2763 (time 1790102490.00 total 30.05 last 98.57) - open/close 3740 (time 1790102500.01 total 40.06 last 97.61) - open/close 4620 (time 1790102510.02 total 50.07 last 87.93) - open/close 5520 (time 1790102520.02 total 60.07 last 89.98) - open/close 6436 (time 1790102530.03 total 70.07 last 91.54) - open/close 7379 (time 1790102540.03 total 80.07 last 94.29) - open/close 8169 (time 1790102550.04 total 90.09 last 78.87) - open/close 9019 (time 1790102560.06 total 100.11 last 84.86) - open/close 9821 (time 1790102570.06 total 110.11 last 80.17) total: 10000 open/close in 112.22 seconds: 89.11 ops/second - unlinked 0 (time 1790102576 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 860 (time 1790102650.97 total 10.00 last 85.98) - open/close 1716 (time 1790102660.97 total 20.01 last 85.54) - open/close 2548 (time 1790102670.98 total 30.02 last 83.14) - open/close 3499 (time 1790102680.99 total 40.02 last 95.05) - open/close 4347 (time 1790102690.99 total 50.02 last 84.80) - open/close 5317 (time 1790102700.99 total 60.03 last 96.94) - open/close 6145 (time 1790102711.01 total 70.04 last 82.70) - open/close 6984 (time 1790102721.01 total 80.04 last 83.90) - open/close 7863 (time 1790102731.01 total 90.05 last 87.84) - open/close 8705 (time 1790102741.03 total 100.06 last 84.09) - open/close 9554 (time 1790102751.04 total 110.07 last 84.81) total: 10000 open/close in 115.32 seconds: 86.71 ops/second - unlinked 0 (time 1790102761 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: kmemleak disabled modules unloaded. pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2