== conf-sanity test 69: replace an OST with the same index ========================================================== 00:09:32 (1783138172) start mds service on oleg349-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg349-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 706 (time 1783138300.15 total 10.03 last 70.42) - open/close 1408 (time 1783138310.16 total 20.03 last 70.14) - open/close 2161 (time 1783138320.17 total 30.04 last 75.23) - open/close 2839 (time 1783138330.17 total 40.05 last 67.77) - open/close 3517 (time 1783138340.17 total 50.05 last 67.80) - open/close 4200 (time 1783138350.19 total 60.06 last 68.19) - open/close 4877 (time 1783138360.19 total 70.07 last 67.67) - open/close 5473 (time 1783138370.20 total 80.08 last 59.54) - open/close 6141 (time 1783138380.21 total 90.08 last 66.75) - open/close 6844 (time 1783138390.21 total 100.09 last 70.28) - open/close 7569 (time 1783138400.23 total 110.10 last 72.39) - open/close 8241 (time 1783138410.24 total 120.11 last 67.14) - open/close 9007 (time 1783138420.25 total 130.12 last 76.53) - open/close 9712 (time 1783138430.26 total 140.13 last 70.41) total: 10000 open/close in 143.97 seconds: 69.46 ops/second - unlinked 0 (time 1783138438 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 692 (time 1783138529.85 total 10.01 last 69.10) - open/close 1397 (time 1783138539.85 total 20.01 last 70.50) - open/close 2090 (time 1783138549.86 total 30.03 last 69.22) - open/close 2812 (time 1783138559.87 total 40.04 last 72.10) - open/close 3590 (time 1783138569.90 total 50.07 last 77.61) - open/close 4363 (time 1783138579.90 total 60.07 last 77.27) - open/close 5054 (time 1783138589.90 total 70.07 last 69.07) - open/close 5833 (time 1783138599.90 total 80.07 last 77.90) - open/close 6585 (time 1783138609.91 total 90.08 last 75.17) - open/close 7382 (time 1783138619.93 total 100.10 last 79.53) - open/close 8120 (time 1783138629.93 total 110.10 last 73.78) - open/close 8837 (time 1783138639.95 total 120.12 last 71.58) - open/close 9586 (time 1783138649.97 total 130.14 last 74.77) total: 10000 open/close in 135.84 seconds: 73.62 ops/second - unlinked 0 (time 1783138660 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 789 (time 1783138750.60 total 10.01 last 78.85) - open/close 1554 (time 1783138760.60 total 20.01 last 76.50) - open/close 2291 (time 1783138770.62 total 30.02 last 73.58) - open/close 3086 (time 1783138780.63 total 40.04 last 79.40) - open/close 3845 (time 1783138790.63 total 50.04 last 75.89) - open/close 4569 (time 1783138800.65 total 60.06 last 72.25) - open/close 5298 (time 1783138810.66 total 70.07 last 72.82) - open/close 5983 (time 1783138820.67 total 80.08 last 68.43) - open/close 6701 (time 1783138830.68 total 90.08 last 71.78) - open/close 7470 (time 1783138840.68 total 100.08 last 76.90) - open/close 8186 (time 1783138850.69 total 110.09 last 71.54) - open/close 8942 (time 1783138860.69 total 120.09 last 75.57) - open/close 9670 (time 1783138870.69 total 130.09 last 72.80) total: 10000 open/close in 134.25 seconds: 74.49 ops/second - unlinked 0 (time 1783138878 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 772 (time 1783138969.56 total 10.00 last 77.17) - open/close 1584 (time 1783138979.56 total 20.01 last 81.14) - open/close 2351 (time 1783138989.56 total 30.01 last 76.69) - open/close 3147 (time 1783138999.58 total 40.02 last 79.51) - open/close 3922 (time 1783139009.58 total 50.03 last 77.46) - open/close 4711 (time 1783139019.59 total 60.04 last 78.84) - open/close 5438 (time 1783139029.61 total 70.06 last 72.56) - open/close 6224 (time 1783139039.63 total 80.08 last 78.45) - open/close 6998 (time 1783139049.64 total 90.09 last 77.31) - open/close 7770 (time 1783139059.65 total 100.10 last 77.10) - open/close 8579 (time 1783139069.66 total 110.11 last 80.84) - open/close 9326 (time 1783139079.67 total 120.11 last 74.66) total: 10000 open/close in 129.20 seconds: 77.40 ops/second - unlinked 0 (time 1783139093 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 768 (time 1783139184.30 total 10.01 last 76.73) - open/close 1507 (time 1783139194.32 total 20.02 last 73.81) - open/close 2271 (time 1783139204.33 total 30.03 last 76.31) - open/close 3039 (time 1783139214.33 total 40.04 last 76.78) - open/close 3814 (time 1783139224.34 total 50.05 last 77.41) - open/close 4523 (time 1783139234.34 total 60.05 last 70.89) - open/close 5258 (time 1783139244.35 total 70.06 last 73.44) - open/close 5994 (time 1783139254.36 total 80.07 last 73.54) - open/close 6736 (time 1783139264.36 total 90.07 last 74.16) - open/close 7511 (time 1783139274.37 total 100.08 last 77.45) - open/close 8281 (time 1783139284.37 total 110.08 last 76.99) - open/close 9028 (time 1783139294.37 total 120.08 last 74.69) - open/close 9817 (time 1783139304.38 total 130.09 last 78.81) total: 10000 open/close in 132.75 seconds: 75.33 ops/second - unlinked 0 (time 1783139311 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 761 (time 1783139403.14 total 10.02 last 75.98) - open/close 1481 (time 1783139413.15 total 20.02 last 71.97) - open/close 2217 (time 1783139423.15 total 30.02 last 73.57) - open/close 2902 (time 1783139433.16 total 40.03 last 68.42) - open/close 3659 (time 1783139443.17 total 50.04 last 75.68) - open/close 4412 (time 1783139453.17 total 60.04 last 75.26) - open/close 5135 (time 1783139463.18 total 70.05 last 72.23) - open/close 5926 (time 1783139473.19 total 80.06 last 79.06) - open/close 6697 (time 1783139483.19 total 90.06 last 77.07) - open/close 7481 (time 1783139493.20 total 100.07 last 78.34) - open/close 8244 (time 1783139503.20 total 110.07 last 76.25) - open/close 9003 (time 1783139513.21 total 120.08 last 75.86) - open/close 9761 (time 1783139523.22 total 130.09 last 75.75) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1783139530 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 673 (time 1783139619.26 total 10.00 last 67.27) - open/close 1264 (time 1783139629.26 total 20.01 last 59.07) - open/close 2017 (time 1783139639.26 total 30.01 last 75.29) - open/close 2848 (time 1783139649.27 total 40.01 last 83.09) - open/close 3595 (time 1783139659.27 total 50.01 last 74.68) - open/close 4303 (time 1783139669.27 total 60.01 last 70.79) - open/close 5046 (time 1783139679.29 total 70.04 last 74.15) - open/close 5718 (time 1783139689.29 total 80.04 last 67.19) - open/close 6417 (time 1783139699.30 total 90.05 last 69.83) - open/close 7124 (time 1783139709.30 total 100.05 last 70.68) - open/close 7862 (time 1783139719.31 total 110.05 last 73.78) - open/close 8568 (time 1783139729.31 total 120.06 last 70.55) - open/close 9335 (time 1783139739.33 total 130.08 last 76.52) total: 10000 open/close in 138.57 seconds: 72.17 ops/second - unlinked 0 (time 1783139752 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 799 (time 1783139839.99 total 10.01 last 79.83) - open/close 1556 (time 1783139850.00 total 20.01 last 75.69) - open/close 2365 (time 1783139860.00 total 30.02 last 80.84) - open/close 3195 (time 1783139870.01 total 40.02 last 82.97) - open/close 4007 (time 1783139880.01 total 50.02 last 81.20) - open/close 4829 (time 1783139890.02 total 60.03 last 82.12) - open/close 5556 (time 1783139900.03 total 70.04 last 72.62) - open/close 6332 (time 1783139910.03 total 80.05 last 77.54) - open/close 7137 (time 1783139920.04 total 90.05 last 80.47) - open/close 7917 (time 1783139930.05 total 100.06 last 77.95) - open/close 8708 (time 1783139940.05 total 110.07 last 79.05) - open/close 9536 (time 1783139950.06 total 120.07 last 82.74) total: 10000 open/close in 126.41 seconds: 79.11 ops/second - unlinked 0 (time 1783139960 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 548 (time 1783140052.21 total 10.01 last 54.74) - open/close 1246 (time 1783140062.22 total 20.02 last 69.76) - open/close 1987 (time 1783140072.24 total 30.03 last 73.99) - open/close 2768 (time 1783140082.25 total 40.05 last 78.00) - open/close 3504 (time 1783140092.26 total 50.06 last 73.50) - open/close 4227 (time 1783140102.26 total 60.06 last 72.29) - open/close 4982 (time 1783140112.28 total 70.07 last 75.42) - open/close 5767 (time 1783140122.28 total 80.08 last 78.44) - open/close 6579 (time 1783140132.29 total 90.08 last 81.15) - open/close 7342 (time 1783140142.29 total 100.08 last 76.30) - open/close 8099 (time 1783140152.30 total 110.10 last 75.62) - open/close 8890 (time 1783140162.30 total 120.10 last 79.09) - open/close 9642 (time 1783140172.30 total 130.10 last 75.19) total: 10000 open/close in 134.32 seconds: 74.45 ops/second - unlinked 0 (time 1783140181 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 740 (time 1783140272.84 total 10.01 last 73.94) - open/close 1449 (time 1783140282.84 total 20.02 last 70.85) - open/close 2234 (time 1783140292.85 total 30.02 last 78.48) - open/close 3005 (time 1783140302.86 total 40.04 last 76.97) - open/close 3769 (time 1783140312.87 total 50.05 last 76.33) - open/close 4598 (time 1783140322.87 total 60.05 last 82.89) - open/close 5402 (time 1783140332.89 total 70.06 last 80.30) - open/close 6128 (time 1783140342.90 total 80.07 last 72.54) - open/close 6921 (time 1783140352.91 total 90.08 last 79.17) - open/close 7702 (time 1783140362.91 total 100.08 last 78.09) - open/close 8466 (time 1783140372.93 total 110.10 last 76.29) - open/close 9204 (time 1783140382.94 total 120.11 last 73.72) - open/close 9973 (time 1783140392.95 total 130.12 last 76.80) total: 10000 open/close in 130.45 seconds: 76.66 ops/second - unlinked 0 (time 1783140398 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.149@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg349-server: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg349-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg349-server: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg349-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.149@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.149@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.149@tcp sys.timeout=20 start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg349-server: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg349-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg349-server: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg349-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2