== conf-sanity test 69: replace an OST with the same index ========================================================== 11:36:56 (1788622616) start mds service on oleg125-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: kmemleak disabled oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg125-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg125-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1009 (time 1788622733.47 total 10.01 last 100.83) - open/close 2009 (time 1788622743.48 total 20.01 last 99.94) - open/close 3044 (time 1788622753.48 total 30.02 last 103.46) - open/close 3932 (time 1788622763.49 total 40.03 last 88.73) - open/close 4896 (time 1788622773.50 total 50.03 last 96.35) - open/close 5579 (time 1788622783.50 total 60.03 last 68.28) - open/close 6324 (time 1788622793.50 total 70.03 last 74.50) - open/close 7184 (time 1788622803.51 total 80.05 last 85.87) - open/close 7973 (time 1788622813.52 total 90.05 last 78.86) - open/close 8836 (time 1788622823.54 total 100.07 last 86.13) - open/close 9635 (time 1788622833.54 total 110.08 last 79.87) total: 10000 open/close in 114.02 seconds: 87.70 ops/second - unlinked 0 (time 1788622842 ; total 1 ; last 1) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 928 (time 1788622923.26 total 10.00 last 92.78) - open/close 1776 (time 1788622933.29 total 20.04 last 84.51) - open/close 2364 (time 1788622943.32 total 30.06 last 58.64) - open/close 2995 (time 1788622953.33 total 40.08 last 63.03) - open/close 3541 (time 1788622963.34 total 50.09 last 54.55) - open/close 4100 (time 1788622973.34 total 60.09 last 55.89) - open/close 4946 (time 1788622983.35 total 70.09 last 84.54) - open/close 5820 (time 1788622993.36 total 80.10 last 87.37) - open/close 6709 (time 1788623003.36 total 90.10 last 88.85) - open/close 7503 (time 1788623013.38 total 100.12 last 79.26) - open/close 8374 (time 1788623023.39 total 110.13 last 87.02) - open/close 9172 (time 1788623033.39 total 120.13 last 79.80) total: 10000 open/close in 129.39 seconds: 77.29 ops/second - unlinked 0 (time 1788623047 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 964 (time 1788623124.68 total 10.01 last 96.29) - open/close 1872 (time 1788623134.69 total 20.02 last 90.72) - open/close 2642 (time 1788623144.69 total 30.02 last 76.98) - open/close 3380 (time 1788623154.70 total 40.04 last 73.71) - open/close 4187 (time 1788623164.71 total 50.04 last 80.66) - open/close 5066 (time 1788623174.71 total 60.05 last 87.82) - open/close 6066 (time 1788623184.73 total 70.06 last 99.84) - open/close 7057 (time 1788623194.74 total 80.07 last 99.06) - open/close 8065 (time 1788623204.74 total 90.07 last 100.77) - open/close 8923 (time 1788623214.75 total 100.08 last 85.71) - open/close 9678 (time 1788623224.75 total 110.09 last 75.46) total: 10000 open/close in 113.81 seconds: 87.87 ops/second - unlinked 0 (time 1788623233 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 838 (time 1788623315.60 total 10.00 last 83.79) - open/close 1407 (time 1788623325.60 total 20.01 last 56.86) - open/close 2145 (time 1788623335.62 total 30.02 last 73.71) - open/close 2822 (time 1788623345.62 total 40.02 last 67.68) - open/close 3581 (time 1788623355.62 total 50.03 last 75.88) - open/close 4462 (time 1788623365.62 total 60.03 last 88.09) - open/close 5355 (time 1788623375.63 total 70.03 last 89.26) - open/close 6126 (time 1788623385.63 total 80.03 last 77.10) - open/close 6940 (time 1788623395.64 total 90.05 last 81.29) - open/close 7670 (time 1788623405.65 total 100.05 last 72.96) - open/close 8624 (time 1788623415.65 total 110.05 last 95.38) - open/close 9529 (time 1788623425.66 total 120.06 last 90.44) total: 10000 open/close in 125.22 seconds: 79.86 ops/second - unlinked 0 (time 1788623435 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 950 (time 1788623520.37 total 10.00 last 94.97) - open/close 1832 (time 1788623530.39 total 20.02 last 88.08) - open/close 2672 (time 1788623540.40 total 30.03 last 83.86) - open/close 3440 (time 1788623550.43 total 40.05 last 76.64) - open/close 4248 (time 1788623560.43 total 50.06 last 80.74) - open/close 4925 (time 1788623570.44 total 60.07 last 67.67) - open/close 5793 (time 1788623580.44 total 70.07 last 86.75) - open/close 6530 (time 1788623590.45 total 80.08 last 73.66) - open/close 7461 (time 1788623600.45 total 90.08 last 93.06) - open/close 8194 (time 1788623610.46 total 100.09 last 73.24) - open/close 9183 (time 1788623620.46 total 110.09 last 98.90) total: 10000 open/close in 118.73 seconds: 84.23 ops/second - unlinked 0 (time 1788623633 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 887 (time 1788623717.95 total 10.01 last 88.65) - open/close 1619 (time 1788623727.96 total 20.01 last 73.14) - open/close 2431 (time 1788623737.98 total 30.04 last 81.01) - open/close 3372 (time 1788623747.99 total 40.04 last 94.08) - open/close 4325 (time 1788623757.99 total 50.04 last 95.26) - open/close 5310 (time 1788623767.99 total 60.05 last 98.44) - open/close 6144 (time 1788623778.01 total 70.07 last 83.26) - open/close 6992 (time 1788623788.01 total 80.07 last 84.78) - open/close 7837 (time 1788623798.03 total 90.09 last 84.33) - open/close 8707 (time 1788623808.04 total 100.09 last 86.98) - open/close 9521 (time 1788623818.05 total 110.10 last 81.30) total: 10000 open/close in 115.83 seconds: 86.33 ops/second - unlinked 0 (time 1788623829 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 832 (time 1788623916.60 total 10.00 last 83.18) - open/close 1684 (time 1788623926.61 total 20.01 last 85.16) - open/close 2504 (time 1788623936.61 total 30.02 last 81.94) - open/close 3288 (time 1788623946.61 total 40.02 last 78.40) - open/close 4240 (time 1788623956.62 total 50.02 last 95.17) - open/close 5093 (time 1788623966.62 total 60.03 last 85.25) - open/close 5949 (time 1788623976.63 total 70.03 last 85.57) - open/close 6487 (time 1788623986.65 total 80.05 last 53.69) - open/close 7120 (time 1788623996.65 total 90.06 last 63.24) - open/close 7909 (time 1788624006.66 total 100.06 last 78.90) - open/close 8781 (time 1788624016.66 total 110.06 last 87.20) - open/close 9539 (time 1788624026.66 total 120.06 last 75.75) total: 10000 open/close in 125.25 seconds: 79.84 ops/second - unlinked 0 (time 1788624037 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 892 (time 1788624120.75 total 10.01 last 89.13) - open/close 1834 (time 1788624130.76 total 20.02 last 94.12) - open/close 2706 (time 1788624140.77 total 30.03 last 87.10) - open/close 3562 (time 1788624150.78 total 40.04 last 85.51) - open/close 4415 (time 1788624160.80 total 50.05 last 85.17) - open/close 5191 (time 1788624170.80 total 60.06 last 77.57) - open/close 6003 (time 1788624180.80 total 70.06 last 81.20) - open/close 6944 (time 1788624190.81 total 80.07 last 93.98) - open/close 7843 (time 1788624200.84 total 90.09 last 89.69) - open/close 8760 (time 1788624210.85 total 100.10 last 91.62) - open/close 9720 (time 1788624220.87 total 110.13 last 95.76) total: 10000 open/close in 113.54 seconds: 88.07 ops/second - unlinked 0 (time 1788624228 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 880 (time 1788624315.09 total 10.01 last 87.95) - open/close 1813 (time 1788624325.09 total 20.01 last 93.28) - open/close 2652 (time 1788624335.09 total 30.01 last 83.88) - open/close 3689 (time 1788624345.10 total 40.01 last 103.66) - open/close 4623 (time 1788624355.11 total 50.02 last 93.32) - open/close 5514 (time 1788624365.11 total 60.03 last 89.08) - open/close 6393 (time 1788624375.12 total 70.04 last 87.81) - open/close 7288 (time 1788624385.13 total 80.05 last 89.40) - open/close 8105 (time 1788624395.14 total 90.05 last 81.65) - open/close 8928 (time 1788624405.14 total 100.05 last 82.28) - open/close 9831 (time 1788624415.14 total 110.06 last 90.28) total: 10000 open/close in 112.11 seconds: 89.20 ops/second - unlinked 0 (time 1788624421 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1025 (time 1788624494.07 total 10.00 last 102.49) - open/close 2003 (time 1788624504.08 total 20.01 last 97.72) - open/close 2967 (time 1788624514.08 total 30.01 last 96.35) - open/close 3878 (time 1788624524.09 total 40.02 last 91.02) - open/close 4797 (time 1788624534.10 total 50.03 last 91.85) - open/close 5790 (time 1788624544.11 total 60.04 last 99.22) - open/close 6779 (time 1788624554.11 total 70.04 last 98.89) - open/close 7675 (time 1788624564.11 total 80.04 last 89.60) - open/close 8647 (time 1788624574.12 total 90.05 last 97.08) - open/close 9653 (time 1788624584.12 total 100.06 last 100.54) total: 10000 open/close in 103.61 seconds: 96.52 ops/second - unlinked 0 (time 1788624591 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: kmemleak disabled modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2