== conf-sanity test 69: replace an OST with the same index ========================================================== 16:40:56 (1788813656) start mds service on oleg411-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931936 Sep 7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: kmemleak disabled oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49535 inodes available. Want 99872. rc=0 - open/close 738 (time 1788813772.67 total 10.01 last 73.74) - open/close 1527 (time 1788813782.68 total 20.01 last 78.86) - open/close 2377 (time 1788813792.68 total 30.02 last 84.94) - open/close 3373 (time 1788813802.70 total 40.03 last 99.45) - open/close 4185 (time 1788813812.70 total 50.04 last 81.17) - open/close 4921 (time 1788813822.71 total 60.05 last 73.54) - open/close 5796 (time 1788813832.71 total 70.05 last 87.47) - open/close 6678 (time 1788813842.72 total 80.05 last 88.16) - open/close 7507 (time 1788813852.72 total 90.06 last 82.89) - open/close 8476 (time 1788813862.73 total 100.06 last 96.82) - open/close 9361 (time 1788813872.73 total 110.07 last 88.45) total: 10000 open/close in 117.06 seconds: 85.43 ops/second - unlinked 0 (time 1788813884 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 912 (time 1788813965.25 total 10.00 last 91.18) - open/close 1753 (time 1788813975.26 total 20.01 last 84.07) - open/close 2543 (time 1788813985.27 total 30.02 last 78.90) - open/close 3342 (time 1788813995.27 total 40.02 last 79.86) - open/close 4107 (time 1788814005.29 total 50.04 last 76.41) - open/close 4840 (time 1788814015.30 total 60.05 last 73.17) - open/close 5639 (time 1788814025.31 total 70.06 last 79.81) - open/close 6500 (time 1788814035.32 total 80.07 last 86.09) - open/close 7438 (time 1788814045.32 total 90.07 last 93.72) - open/close 8259 (time 1788814055.33 total 100.08 last 82.09) - open/close 9201 (time 1788814065.34 total 110.09 last 94.09) total: 10000 open/close in 118.99 seconds: 84.04 ops/second - unlinked 0 (time 1788814078 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 860 (time 1788814162.40 total 10.00 last 85.96) - open/close 1748 (time 1788814172.41 total 20.01 last 88.78) - open/close 2615 (time 1788814182.41 total 30.01 last 86.64) - open/close 3506 (time 1788814192.41 total 40.02 last 89.07) - open/close 4371 (time 1788814202.43 total 50.03 last 86.41) - open/close 5252 (time 1788814212.44 total 60.04 last 87.97) - open/close 6161 (time 1788814222.44 total 70.05 last 90.86) - open/close 7012 (time 1788814232.45 total 80.05 last 85.04) - open/close 7829 (time 1788814242.45 total 90.05 last 81.68) - open/close 8655 (time 1788814252.46 total 100.06 last 82.52) - open/close 9468 (time 1788814262.47 total 110.08 last 81.21) total: 10000 open/close in 117.21 seconds: 85.32 ops/second - unlinked 0 (time 1788814273 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 897 (time 1788814352.12 total 10.01 last 89.64) - open/close 1747 (time 1788814362.12 total 20.02 last 84.92) - open/close 2594 (time 1788814372.13 total 30.02 last 84.67) - open/close 3447 (time 1788814382.14 total 40.03 last 85.22) - open/close 4296 (time 1788814392.14 total 50.03 last 84.90) - open/close 5157 (time 1788814402.16 total 60.05 last 85.94) - open/close 6038 (time 1788814412.16 total 70.05 last 88.07) - open/close 6913 (time 1788814422.17 total 80.06 last 87.43) - open/close 7744 (time 1788814432.17 total 90.06 last 83.06) - open/close 8663 (time 1788814442.18 total 100.07 last 91.86) - open/close 9486 (time 1788814452.18 total 110.07 last 82.26) total: 10000 open/close in 116.19 seconds: 86.06 ops/second - unlinked 0 (time 1788814461 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 749 (time 1788814542.67 total 10.02 last 74.75) - open/close 1555 (time 1788814552.67 total 20.02 last 80.56) - open/close 2389 (time 1788814562.68 total 30.03 last 83.34) - open/close 3264 (time 1788814572.68 total 40.04 last 87.47) - open/close 4122 (time 1788814582.69 total 50.04 last 85.77) - open/close 4920 (time 1788814592.70 total 60.05 last 79.73) - open/close 5788 (time 1788814602.70 total 70.05 last 86.79) - open/close 6643 (time 1788814612.70 total 80.05 last 85.49) - open/close 7531 (time 1788814622.70 total 90.05 last 88.76) - open/close 8399 (time 1788814632.71 total 100.06 last 86.76) - open/close 9264 (time 1788814642.72 total 110.07 last 86.40) total: 10000 open/close in 118.49 seconds: 84.39 ops/second - unlinked 0 (time 1788814654 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 891 (time 1788814727.99 total 10.01 last 89.04) - open/close 1778 (time 1788814738.01 total 20.02 last 88.58) - open/close 2649 (time 1788814748.01 total 30.03 last 87.03) - open/close 3522 (time 1788814758.03 total 40.04 last 87.19) - open/close 4333 (time 1788814768.04 total 50.05 last 81.02) - open/close 5199 (time 1788814778.05 total 60.06 last 86.53) - open/close 6046 (time 1788814788.05 total 70.06 last 84.67) - open/close 6888 (time 1788814798.05 total 80.07 last 84.18) - open/close 7736 (time 1788814808.05 total 90.07 last 84.80) - open/close 8599 (time 1788814818.06 total 100.07 last 86.24) - open/close 9436 (time 1788814828.06 total 110.08 last 83.67) total: 10000 open/close in 116.80 seconds: 85.61 ops/second - unlinked 0 (time 1788814838 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 879 (time 1788814916.91 total 10.00 last 87.87) - open/close 1769 (time 1788814926.92 total 20.01 last 88.95) - open/close 2618 (time 1788814936.92 total 30.01 last 84.85) - open/close 3465 (time 1788814946.93 total 40.02 last 84.63) - open/close 4344 (time 1788814956.94 total 50.03 last 87.84) - open/close 5148 (time 1788814966.95 total 60.04 last 80.33) - open/close 6043 (time 1788814976.96 total 70.05 last 89.42) - open/close 6918 (time 1788814986.96 total 80.05 last 87.49) - open/close 7810 (time 1788814996.97 total 90.06 last 89.12) - open/close 8736 (time 1788815006.97 total 100.07 last 92.53) - open/close 9597 (time 1788815016.98 total 110.07 last 86.08) total: 10000 open/close in 114.78 seconds: 87.13 ops/second - unlinked 0 (time 1788815025 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 867 (time 1788815102.21 total 10.01 last 86.65) - open/close 1738 (time 1788815112.21 total 20.01 last 87.06) - open/close 2530 (time 1788815122.21 total 30.01 last 79.19) - open/close 3400 (time 1788815132.22 total 40.02 last 86.92) - open/close 4240 (time 1788815142.24 total 50.04 last 83.87) - open/close 5075 (time 1788815152.25 total 60.05 last 83.40) - open/close 5920 (time 1788815162.27 total 70.06 last 84.36) - open/close 6820 (time 1788815172.27 total 80.07 last 89.92) - open/close 7722 (time 1788815182.28 total 90.08 last 90.14) - open/close 8602 (time 1788815192.29 total 100.09 last 87.95) - open/close 9481 (time 1788815202.29 total 110.09 last 87.85) total: 10000 open/close in 115.82 seconds: 86.34 ops/second - unlinked 0 (time 1788815211 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 831 (time 1788815284.60 total 10.00 last 83.09) - open/close 1744 (time 1788815294.61 total 20.01 last 91.21) - open/close 2639 (time 1788815304.61 total 30.01 last 89.49) - open/close 3545 (time 1788815314.61 total 40.01 last 90.58) - open/close 4438 (time 1788815324.62 total 50.02 last 89.25) - open/close 5339 (time 1788815334.62 total 60.02 last 90.07) - open/close 6292 (time 1788815344.62 total 70.02 last 95.30) - open/close 7214 (time 1788815354.62 total 80.02 last 92.19) - open/close 8132 (time 1788815364.62 total 90.03 last 91.78) - open/close 9032 (time 1788815374.63 total 100.03 last 89.97) - open/close 9782 (time 1788815384.63 total 110.03 last 74.96) total: 10000 open/close in 112.61 seconds: 88.80 ops/second - unlinked 0 (time 1788815390 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 772 (time 1788815469.74 total 10.01 last 77.15) - open/close 1565 (time 1788815479.75 total 20.02 last 79.19) - open/close 2366 (time 1788815489.76 total 30.03 last 80.00) - open/close 3161 (time 1788815499.77 total 40.04 last 79.44) - open/close 4020 (time 1788815509.77 total 50.04 last 85.90) - open/close 4962 (time 1788815519.78 total 60.05 last 94.13) - open/close 5943 (time 1788815529.78 total 70.05 last 98.05) - open/close 6811 (time 1788815539.79 total 80.06 last 86.71) - open/close 7705 (time 1788815549.80 total 90.07 last 89.36) - open/close 8567 (time 1788815559.80 total 100.07 last 86.16) - open/close 9460 (time 1788815569.81 total 110.08 last 89.27) total: 10000 open/close in 116.39 seconds: 85.91 ops/second - unlinked 0 (time 1788815579 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.111@tcp sys.timeout=20 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg411-server: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg411-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: kmemleak disabled modules unloaded. pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2