== conf-sanity test 69: replace an OST with the same index ========================================================== 03:11:56 (1785309116) start mds service on oleg655-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg655-server' oleg655-server: oleg655-server.virtnet: executing load_modules_local oleg655-server: Loading modules from /home/green/git/lustre-release/lustre oleg655-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg655-server: detected 4 online CPUs by sysfs oleg655-server: MODOPTS_LIBCFS= oleg655-server: Force libcfs to create 2 CPU partitions oleg655-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg655-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg655-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg655-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg655-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 906 (time 1785309221.53 total 10.00 last 90.60) - open/close 1785 (time 1785309231.54 total 20.01 last 87.82) - open/close 2603 (time 1785309241.54 total 30.01 last 81.76) - open/close 3338 (time 1785309251.56 total 40.03 last 73.38) - open/close 4126 (time 1785309261.56 total 50.04 last 78.75) - open/close 4903 (time 1785309271.58 total 60.05 last 77.59) - open/close 5660 (time 1785309281.58 total 70.05 last 75.69) - open/close 6576 (time 1785309291.60 total 80.07 last 91.43) - open/close 7277 (time 1785309301.60 total 90.08 last 70.05) - open/close 8034 (time 1785309311.62 total 100.10 last 75.56) - open/close 8784 (time 1785309321.63 total 110.10 last 74.95) - open/close 9671 (time 1785309331.63 total 120.11 last 88.66) total: 10000 open/close in 123.47 seconds: 80.99 ops/second - unlinked 0 (time 1785309338 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 751 (time 1785309422.72 total 10.00 last 75.07) - open/close 1294 (time 1785309432.74 total 20.02 last 54.20) - open/close 1985 (time 1785309442.74 total 30.03 last 69.06) - open/close 2723 (time 1785309452.75 total 40.03 last 73.76) - open/close 3616 (time 1785309462.76 total 50.05 last 89.18) - open/close 4362 (time 1785309472.78 total 60.06 last 74.48) - open/close 5002 (time 1785309482.78 total 70.07 last 63.95) - open/close 5714 (time 1785309492.79 total 80.08 last 71.14) - open/close 6508 (time 1785309502.79 total 90.08 last 79.40) - open/close 7337 (time 1785309512.81 total 100.09 last 82.78) - open/close 8182 (time 1785309522.81 total 110.10 last 84.45) - open/close 8918 (time 1785309532.81 total 120.10 last 73.59) - open/close 9654 (time 1785309542.82 total 130.11 last 73.56) total: 10000 open/close in 134.00 seconds: 74.63 ops/second - unlinked 0 (time 1785309551 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 754 (time 1785309637.66 total 10.00 last 75.37) - open/close 1601 (time 1785309647.69 total 20.03 last 84.44) - open/close 2340 (time 1785309657.70 total 30.05 last 73.83) - open/close 3081 (time 1785309667.71 total 40.05 last 74.04) - open/close 3754 (time 1785309677.72 total 50.06 last 67.23) - open/close 4482 (time 1785309687.74 total 60.08 last 72.66) - open/close 5240 (time 1785309697.75 total 70.09 last 75.74) - open/close 6055 (time 1785309707.75 total 80.09 last 81.47) - open/close 6872 (time 1785309717.76 total 90.10 last 81.63) - open/close 7632 (time 1785309727.77 total 100.11 last 75.92) - open/close 8360 (time 1785309737.77 total 110.12 last 72.76) - open/close 9182 (time 1785309747.78 total 120.12 last 82.19) total: 10000 open/close in 129.88 seconds: 76.99 ops/second - unlinked 0 (time 1785309761 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 531 (time 1785309847.27 total 10.01 last 53.07) - open/close 1084 (time 1785309857.27 total 20.01 last 55.30) - open/close 1751 (time 1785309867.28 total 30.01 last 66.69) - open/close 2540 (time 1785309877.28 total 40.01 last 78.89) - open/close 3337 (time 1785309887.28 total 50.01 last 79.67) - open/close 4013 (time 1785309897.31 total 60.04 last 67.40) - open/close 4577 (time 1785309907.31 total 70.04 last 56.40) - open/close 5262 (time 1785309917.32 total 80.05 last 68.43) - open/close 5890 (time 1785309927.32 total 90.05 last 62.79) - open/close 6646 (time 1785309937.33 total 100.06 last 75.58) - open/close 7385 (time 1785309947.34 total 110.08 last 73.76) - open/close 7986 (time 1785309957.35 total 120.08 last 60.08) - open/close 8787 (time 1785309967.36 total 130.09 last 80.02) - open/close 9484 (time 1785309977.38 total 140.11 last 69.55) total: 10000 open/close in 146.65 seconds: 68.19 ops/second - unlinked 0 (time 1785309987 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 782 (time 1785310075.56 total 10.00 last 78.20) - open/close 1541 (time 1785310085.57 total 20.01 last 75.84) - open/close 2275 (time 1785310095.58 total 30.01 last 73.36) - open/close 2997 (time 1785310105.58 total 40.02 last 72.17) - open/close 3778 (time 1785310115.58 total 50.02 last 78.07) - open/close 4570 (time 1785310125.59 total 60.03 last 79.16) - open/close 5330 (time 1785310135.59 total 70.03 last 75.95) - open/close 6145 (time 1785310145.60 total 80.03 last 81.49) - open/close 6932 (time 1785310155.61 total 90.04 last 78.63) - open/close 7745 (time 1785310165.62 total 100.05 last 81.21) - open/close 8514 (time 1785310175.63 total 110.07 last 76.82) - open/close 9414 (time 1785310185.63 total 120.07 last 89.96) total: 10000 open/close in 127.10 seconds: 78.68 ops/second - unlinked 0 (time 1785310196 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 756 (time 1785310277.14 total 10.00 last 75.59) - open/close 1453 (time 1785310287.18 total 20.04 last 69.47) - open/close 2177 (time 1785310297.18 total 30.04 last 72.37) - open/close 2974 (time 1785310307.18 total 40.04 last 79.69) - open/close 3798 (time 1785310317.19 total 50.05 last 82.36) - open/close 4529 (time 1785310327.19 total 60.05 last 73.04) - open/close 5377 (time 1785310337.20 total 70.06 last 84.77) - open/close 6242 (time 1785310347.20 total 80.06 last 86.47) - open/close 7087 (time 1785310357.22 total 90.08 last 84.37) - open/close 7936 (time 1785310367.22 total 100.08 last 84.84) - open/close 8804 (time 1785310377.23 total 110.08 last 86.78) - open/close 9656 (time 1785310387.23 total 120.09 last 85.15) total: 10000 open/close in 124.27 seconds: 80.47 ops/second - unlinked 0 (time 1785310394 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 788 (time 1785310478.50 total 10.01 last 78.75) - open/close 1552 (time 1785310488.51 total 20.02 last 76.30) - open/close 2293 (time 1785310498.51 total 30.02 last 74.06) - open/close 3154 (time 1785310508.52 total 40.03 last 86.09) - open/close 3903 (time 1785310518.52 total 50.03 last 74.86) - open/close 4624 (time 1785310528.53 total 60.04 last 72.04) - open/close 5417 (time 1785310538.54 total 70.05 last 79.24) - open/close 6241 (time 1785310548.54 total 80.05 last 82.38) - open/close 7030 (time 1785310558.54 total 90.05 last 78.86) - open/close 7821 (time 1785310568.55 total 100.06 last 79.08) - open/close 8639 (time 1785310578.56 total 110.07 last 81.72) - open/close 9475 (time 1785310588.57 total 120.08 last 83.50) total: 10000 open/close in 126.49 seconds: 79.06 ops/second - unlinked 0 (time 1785310598 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 763 (time 1785310684.11 total 10.01 last 76.25) - open/close 1489 (time 1785310694.12 total 20.02 last 72.53) - open/close 2306 (time 1785310704.13 total 30.03 last 81.63) - open/close 3098 (time 1785310714.14 total 40.04 last 79.10) - open/close 3885 (time 1785310724.16 total 50.06 last 78.56) - open/close 4591 (time 1785310734.16 total 60.06 last 70.58) - open/close 5335 (time 1785310744.16 total 70.06 last 74.37) - open/close 6144 (time 1785310754.18 total 80.07 last 80.80) - open/close 7024 (time 1785310764.18 total 90.08 last 87.93) - open/close 7823 (time 1785310774.20 total 100.10 last 79.77) - open/close 8570 (time 1785310784.21 total 110.10 last 74.66) - open/close 9349 (time 1785310794.21 total 120.11 last 77.87) total: 10000 open/close in 128.97 seconds: 77.54 ops/second - unlinked 0 (time 1785310807 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 843 (time 1785310892.29 total 10.02 last 84.13) - open/close 1664 (time 1785310902.30 total 20.03 last 82.01) - open/close 2505 (time 1785310912.31 total 30.04 last 84.01) - open/close 3364 (time 1785310922.32 total 40.05 last 85.84) - open/close 4196 (time 1785310932.32 total 50.05 last 83.19) - open/close 5046 (time 1785310942.32 total 60.06 last 84.94) - open/close 5862 (time 1785310952.33 total 70.06 last 81.59) - open/close 6712 (time 1785310962.33 total 80.06 last 84.99) - open/close 7531 (time 1785310972.33 total 90.06 last 81.86) - open/close 8299 (time 1785310982.34 total 100.07 last 76.75) - open/close 9138 (time 1785310992.34 total 110.07 last 83.90) - open/close 9932 (time 1785311002.34 total 120.08 last 79.36) total: 10000 open/close in 120.89 seconds: 82.72 ops/second - unlinked 0 (time 1785311006 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 880 (time 1785311086.19 total 10.00 last 87.96) - open/close 1747 (time 1785311096.20 total 20.01 last 86.62) - open/close 2657 (time 1785311106.20 total 30.02 last 90.98) - open/close 3533 (time 1785311116.21 total 40.03 last 87.49) - open/close 4425 (time 1785311126.23 total 50.04 last 89.07) - open/close 5421 (time 1785311136.23 total 60.05 last 99.54) - open/close 6465 (time 1785311146.24 total 70.06 last 104.34) - open/close 7477 (time 1785311156.24 total 80.06 last 101.15) - open/close 8414 (time 1785311166.24 total 90.06 last 93.70) - open/close 9436 (time 1785311176.24 total 100.06 last 102.20) total: 10000 open/close in 105.67 seconds: 94.63 ops/second - unlinked 0 (time 1785311184 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg655-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg655-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg655-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg655-server: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg655-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg655-server: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg655-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre On OST0, 19918 used inodes rc=0 pdsh@oleg655-client: oleg655-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg655-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg655-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.155@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.155@tcp sys.timeout=20 start ost1 service on oleg655-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg655-server: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg655-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg655-server: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg655-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg655-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg655-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server stop mds service on oleg655-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg655-server stop mds service on oleg655-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg655-server unloading modules via unload_modules_local on: 'oleg655-server' oleg655-server: oleg655-server.virtnet: executing unload_modules_local oleg655-server: modules unloaded. pdsh@oleg655-client: oleg655-client: ssh exited with exit code 2 pdsh@oleg655-client: oleg655-server: ssh exited with exit code 2 pdsh@oleg655-client: oleg655-client: ssh exited with exit code 2 pdsh@oleg655-client: oleg655-server: ssh exited with exit code 2