== conf-sanity test 69: replace an OST with the same index ========================================================== 02:23:54 (1786947834) start mds service on oleg107-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 644 (time 1786947954.36 total 10.00 last 64.37) - open/close 1321 (time 1786947964.37 total 20.01 last 67.69) - open/close 2048 (time 1786947974.39 total 30.03 last 72.55) - open/close 2638 (time 1786947984.40 total 40.04 last 58.94) - open/close 3361 (time 1786947994.40 total 50.04 last 72.26) - open/close 4050 (time 1786948004.41 total 60.05 last 68.83) - open/close 4822 (time 1786948014.41 total 70.05 last 77.18) - open/close 5560 (time 1786948024.41 total 80.05 last 73.79) - open/close 6294 (time 1786948034.42 total 90.06 last 73.32) - open/close 6874 (time 1786948044.43 total 100.07 last 57.96) - open/close 7539 (time 1786948054.44 total 110.08 last 66.44) - open/close 8172 (time 1786948064.44 total 120.08 last 63.29) - open/close 8820 (time 1786948074.45 total 130.09 last 64.76) - open/close 9497 (time 1786948084.47 total 140.12 last 67.53) total: 10000 open/close in 147.14 seconds: 67.96 ops/second - unlinked 0 (time 1786948096 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 619 (time 1786948206.72 total 10.00 last 61.90) - open/close 1232 (time 1786948216.73 total 20.01 last 61.25) - open/close 1926 (time 1786948226.75 total 30.03 last 69.23) - open/close 2601 (time 1786948236.75 total 40.03 last 67.48) - open/close 3196 (time 1786948246.76 total 50.04 last 59.44) - open/close 3776 (time 1786948256.81 total 60.09 last 57.73) - open/close 4385 (time 1786948266.82 total 70.10 last 60.83) - open/close 5192 (time 1786948276.83 total 80.11 last 80.66) - open/close 6074 (time 1786948286.83 total 90.11 last 88.14) - open/close 6904 (time 1786948296.84 total 100.12 last 82.95) - open/close 7604 (time 1786948306.85 total 110.13 last 69.91) - open/close 8334 (time 1786948316.87 total 120.15 last 72.90) - open/close 9050 (time 1786948326.87 total 130.15 last 71.55) - open/close 9913 (time 1786948336.88 total 140.16 last 86.23) total: 10000 open/close in 141.07 seconds: 70.89 ops/second - unlinked 0 (time 1786948343 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 837 (time 1786948428.81 total 10.00 last 83.70) - open/close 1808 (time 1786948438.82 total 20.01 last 97.04) - open/close 2666 (time 1786948448.83 total 30.01 last 85.74) - open/close 3395 (time 1786948458.84 total 40.03 last 72.81) - open/close 4094 (time 1786948468.85 total 50.03 last 69.85) - open/close 4602 (time 1786948478.86 total 60.04 last 50.75) - open/close 5124 (time 1786948488.86 total 70.05 last 52.18) - open/close 5802 (time 1786948498.92 total 80.10 last 67.45) - open/close 6528 (time 1786948508.92 total 90.11 last 72.55) - open/close 7448 (time 1786948518.92 total 100.11 last 92.00) - open/close 8346 (time 1786948528.93 total 110.12 last 89.73) - open/close 9081 (time 1786948538.93 total 120.12 last 73.49) - open/close 9791 (time 1786948548.93 total 130.12 last 70.99) total: 10000 open/close in 133.46 seconds: 74.93 ops/second - unlinked 0 (time 1786948556 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 939 (time 1786948648.68 total 10.01 last 93.84) - open/close 1911 (time 1786948658.68 total 20.01 last 97.19) - open/close 2843 (time 1786948668.69 total 30.01 last 93.14) - open/close 3745 (time 1786948678.69 total 40.01 last 90.19) - open/close 4559 (time 1786948688.69 total 50.02 last 81.37) - open/close 5373 (time 1786948698.70 total 60.02 last 81.36) - open/close 6214 (time 1786948708.70 total 70.03 last 84.06) - open/close 7043 (time 1786948718.71 total 80.04 last 82.83) - open/close 7798 (time 1786948728.71 total 90.04 last 75.47) - open/close 8630 (time 1786948738.71 total 100.04 last 83.19) - open/close 9434 (time 1786948748.72 total 110.05 last 80.35) total: 10000 open/close in 116.55 seconds: 85.80 ops/second - unlinked 0 (time 1786948758 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 856 (time 1786948848.76 total 10.01 last 85.49) - open/close 1658 (time 1786948858.77 total 20.02 last 80.18) - open/close 2420 (time 1786948868.77 total 30.02 last 76.19) - open/close 3230 (time 1786948878.78 total 40.03 last 80.90) - open/close 4061 (time 1786948888.80 total 50.05 last 82.97) - open/close 4852 (time 1786948898.80 total 60.05 last 79.09) - open/close 5693 (time 1786948908.80 total 70.05 last 84.07) - open/close 6435 (time 1786948918.82 total 80.07 last 74.04) - open/close 7212 (time 1786948928.83 total 90.08 last 77.63) - open/close 7998 (time 1786948938.83 total 100.08 last 78.60) - open/close 8682 (time 1786948948.85 total 110.10 last 68.25) - open/close 9375 (time 1786948958.86 total 120.11 last 69.22) - open/close 9989 (time 1786948968.90 total 130.15 last 61.15) total: 10000 open/close in 130.35 seconds: 76.72 ops/second - unlinked 0 (time 1786948974 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 738 (time 1786949061.51 total 10.00 last 73.77) - open/close 1478 (time 1786949071.53 total 20.02 last 73.90) - open/close 2130 (time 1786949081.53 total 30.02 last 65.16) - open/close 2899 (time 1786949091.54 total 40.03 last 76.84) - open/close 3681 (time 1786949101.54 total 50.04 last 78.17) - open/close 4525 (time 1786949111.55 total 60.04 last 84.35) - open/close 5311 (time 1786949121.58 total 70.07 last 78.40) - open/close 6133 (time 1786949131.58 total 80.07 last 82.15) - open/close 6909 (time 1786949141.60 total 90.09 last 77.47) - open/close 7628 (time 1786949151.60 total 100.09 last 71.90) - open/close 8521 (time 1786949161.60 total 110.10 last 89.27) - open/close 9320 (time 1786949171.61 total 120.10 last 79.85) total: 10000 open/close in 129.12 seconds: 77.45 ops/second - unlinked 0 (time 1786949184 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 893 (time 1786949269.98 total 10.01 last 89.22) - open/close 1755 (time 1786949279.99 total 20.02 last 86.13) - open/close 2574 (time 1786949290.00 total 30.03 last 81.83) - open/close 3365 (time 1786949300.00 total 40.03 last 79.08) - open/close 4244 (time 1786949310.00 total 50.03 last 87.89) - open/close 5054 (time 1786949320.00 total 60.03 last 80.99) - open/close 5933 (time 1786949330.01 total 70.04 last 87.81) - open/close 6878 (time 1786949340.01 total 80.04 last 94.48) - open/close 7760 (time 1786949350.01 total 90.04 last 88.20) - open/close 8600 (time 1786949360.02 total 100.05 last 83.94) - open/close 9439 (time 1786949370.03 total 110.06 last 83.84) total: 10000 open/close in 116.18 seconds: 86.07 ops/second - unlinked 0 (time 1786949380 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 913 (time 1786949462.90 total 10.01 last 91.23) - open/close 1856 (time 1786949472.91 total 20.02 last 94.22) - open/close 2847 (time 1786949482.92 total 30.03 last 98.99) - open/close 3792 (time 1786949492.92 total 40.03 last 94.45) - open/close 4739 (time 1786949502.92 total 50.03 last 94.68) - open/close 5783 (time 1786949512.94 total 60.05 last 104.25) - open/close 6667 (time 1786949522.94 total 70.05 last 88.36) - open/close 7569 (time 1786949532.95 total 80.06 last 90.16) - open/close 8448 (time 1786949542.95 total 90.06 last 87.84) - open/close 9380 (time 1786949552.96 total 100.07 last 93.16) total: 10000 open/close in 107.25 seconds: 93.24 ops/second - unlinked 0 (time 1786949563 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 889 (time 1786949638.34 total 10.01 last 88.84) - open/close 1782 (time 1786949648.35 total 20.01 last 89.29) - open/close 2762 (time 1786949658.35 total 30.01 last 97.98) - open/close 3724 (time 1786949668.35 total 40.01 last 96.16) - open/close 4652 (time 1786949678.36 total 50.02 last 92.73) - open/close 5449 (time 1786949688.37 total 60.04 last 79.58) - open/close 5781 (time 1786949698.38 total 70.04 last 33.19) - open/close 6226 (time 1786949708.38 total 80.04 last 44.49) - open/close 6730 (time 1786949718.39 total 90.05 last 50.34) - open/close 7345 (time 1786949728.39 total 100.05 last 61.50) - open/close 8083 (time 1786949738.40 total 110.06 last 73.74) - open/close 8738 (time 1786949748.41 total 120.07 last 65.44) - open/close 9415 (time 1786949758.42 total 130.08 last 67.64) total: 10000 open/close in 137.72 seconds: 72.61 ops/second - unlinked 0 (time 1786949770 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 782 (time 1786949857.76 total 10.02 last 78.07) - open/close 1398 (time 1786949867.77 total 20.02 last 61.57) - open/close 2226 (time 1786949877.78 total 30.04 last 82.69) - open/close 2928 (time 1786949887.79 total 40.05 last 70.12) - open/close 3775 (time 1786949897.80 total 50.05 last 84.67) - open/close 4508 (time 1786949907.80 total 60.05 last 73.29) - open/close 5149 (time 1786949917.80 total 70.05 last 64.09) - open/close 5836 (time 1786949927.80 total 80.05 last 68.69) - open/close 6523 (time 1786949937.81 total 90.07 last 68.60) - open/close 7312 (time 1786949947.82 total 100.08 last 78.84) - open/close 8077 (time 1786949957.84 total 110.10 last 76.33) - open/close 8788 (time 1786949967.85 total 120.10 last 71.07) - open/close 9492 (time 1786949977.86 total 130.11 last 70.34) total: 10000 open/close in 137.94 seconds: 72.50 ops/second - unlinked 0 (time 1786949991 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: modules unloaded. pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2