== conf-sanity test 69: replace an OST with the same index ========================================================== 02:32:03 (1771399923) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg301-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 741 (time 1771400044.61 total 10.01 last 74.05) - open/close 1530 (time 1771400054.62 total 20.01 last 78.84) - open/close 2244 (time 1771400064.63 total 30.02 last 71.35) - open/close 3023 (time 1771400074.64 total 40.03 last 77.83) - open/close 3686 (time 1771400084.64 total 50.04 last 66.27) - open/close 4337 (time 1771400094.65 total 60.04 last 65.05) - open/close 5062 (time 1771400104.68 total 70.07 last 72.32) - open/close 5627 (time 1771400114.68 total 80.07 last 56.49) - open/close 6099 (time 1771400124.69 total 90.08 last 47.16) - open/close 6656 (time 1771400134.69 total 100.08 last 55.70) - open/close 7192 (time 1771400144.69 total 110.08 last 53.59) - open/close 7912 (time 1771400154.70 total 120.09 last 71.91) - open/close 8624 (time 1771400164.70 total 130.09 last 71.20) - open/close 9469 (time 1771400174.71 total 140.11 last 84.38) total: 10000 open/close in 147.00 seconds: 68.03 ops/second - unlinked 0 (time 1771400186 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 760 (time 1771400280.73 total 10.00 last 75.98) - open/close 1664 (time 1771400290.74 total 20.01 last 90.34) - open/close 2474 (time 1771400300.74 total 30.01 last 80.98) - open/close 3300 (time 1771400310.75 total 40.02 last 82.55) - open/close 4277 (time 1771400320.76 total 50.03 last 97.61) - open/close 5069 (time 1771400330.76 total 60.03 last 79.18) - open/close 5933 (time 1771400340.77 total 70.04 last 86.30) - open/close 6728 (time 1771400350.80 total 80.08 last 79.23) - open/close 7420 (time 1771400360.81 total 90.09 last 69.13) - open/close 8161 (time 1771400370.82 total 100.10 last 74.04) - open/close 8880 (time 1771400380.82 total 110.10 last 71.89) - open/close 9672 (time 1771400390.83 total 120.10 last 79.19) total: 10000 open/close in 123.71 seconds: 80.84 ops/second - unlinked 0 (time 1771400398 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 744 (time 1771400493.10 total 10.02 last 74.28) - open/close 1580 (time 1771400503.10 total 20.02 last 83.57) - open/close 2400 (time 1771400513.11 total 30.02 last 81.97) - open/close 3097 (time 1771400523.12 total 40.04 last 69.60) - open/close 3887 (time 1771400533.13 total 50.05 last 78.90) - open/close 4701 (time 1771400543.14 total 60.05 last 81.38) - open/close 5419 (time 1771400553.15 total 70.06 last 71.74) - open/close 6170 (time 1771400563.16 total 80.08 last 74.96) - open/close 6865 (time 1771400573.17 total 90.08 last 69.47) - open/close 7520 (time 1771400583.18 total 100.10 last 65.39) - open/close 8243 (time 1771400593.19 total 110.10 last 72.29) - open/close 9062 (time 1771400603.19 total 120.11 last 81.87) - open/close 9892 (time 1771400613.20 total 130.11 last 82.93) total: 10000 open/close in 131.53 seconds: 76.03 ops/second - unlinked 0 (time 1771400619 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 710 (time 1771400712.14 total 10.01 last 70.93) - open/close 1331 (time 1771400722.14 total 20.02 last 62.07) - open/close 2091 (time 1771400732.15 total 30.02 last 75.94) - open/close 2805 (time 1771400742.17 total 40.04 last 71.27) - open/close 3515 (time 1771400752.17 total 50.04 last 70.99) - open/close 4340 (time 1771400762.18 total 60.05 last 82.45) - open/close 5045 (time 1771400772.18 total 70.05 last 70.47) - open/close 5741 (time 1771400782.18 total 80.05 last 69.59) - open/close 6449 (time 1771400792.20 total 90.07 last 70.69) - open/close 7093 (time 1771400802.21 total 100.08 last 64.30) - open/close 7678 (time 1771400812.22 total 110.10 last 58.42) - open/close 8526 (time 1771400822.23 total 120.11 last 84.74) - open/close 9198 (time 1771400832.23 total 130.11 last 67.18) - open/close 9882 (time 1771400842.25 total 140.12 last 68.31) total: 10000 open/close in 141.69 seconds: 70.57 ops/second - unlinked 0 (time 1771400849 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 767 (time 1771400938.44 total 10.00 last 76.68) - open/close 1479 (time 1771400948.44 total 20.01 last 71.17) - open/close 2203 (time 1771400958.45 total 30.01 last 72.36) - open/close 3007 (time 1771400968.47 total 40.03 last 80.22) - open/close 3689 (time 1771400978.47 total 50.04 last 68.18) - open/close 4426 (time 1771400988.48 total 60.04 last 73.66) - open/close 5335 (time 1771400998.49 total 70.05 last 90.79) - open/close 6001 (time 1771401008.49 total 80.06 last 66.59) - open/close 6688 (time 1771401018.50 total 90.07 last 68.62) - open/close 7159 (time 1771401028.52 total 100.08 last 47.03) - open/close 7803 (time 1771401038.52 total 110.09 last 64.38) - open/close 8512 (time 1771401048.53 total 120.10 last 70.82) - open/close 9130 (time 1771401058.54 total 130.10 last 61.78) - open/close 9944 (time 1771401068.54 total 140.10 last 81.37) total: 10000 open/close in 140.77 seconds: 71.04 ops/second - unlinked 0 (time 1771401074 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 767 (time 1771401170.34 total 10.01 last 76.61) - open/close 1483 (time 1771401180.35 total 20.02 last 71.55) - open/close 2185 (time 1771401190.35 total 30.02 last 70.20) - open/close 3036 (time 1771401200.36 total 40.03 last 85.01) - open/close 3885 (time 1771401210.36 total 50.04 last 84.84) - open/close 4641 (time 1771401220.37 total 60.04 last 75.59) - open/close 5357 (time 1771401230.39 total 70.06 last 71.45) - open/close 5978 (time 1771401240.39 total 80.06 last 62.09) - open/close 6803 (time 1771401250.40 total 90.07 last 82.40) - open/close 7528 (time 1771401260.40 total 100.07 last 72.48) - open/close 8345 (time 1771401270.41 total 110.08 last 81.65) - open/close 9194 (time 1771401280.42 total 120.09 last 84.83) - open/close 9942 (time 1771401290.42 total 130.09 last 74.77) total: 10000 open/close in 130.71 seconds: 76.51 ops/second - unlinked 0 (time 1771401295 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 808 (time 1771401388.86 total 10.01 last 80.74) - open/close 1593 (time 1771401398.86 total 20.01 last 78.49) - open/close 2452 (time 1771401408.87 total 30.02 last 85.83) - open/close 3352 (time 1771401418.89 total 40.04 last 89.79) - open/close 4194 (time 1771401428.89 total 50.04 last 84.18) - open/close 5091 (time 1771401438.90 total 60.05 last 89.62) - open/close 5947 (time 1771401448.90 total 70.05 last 85.58) - open/close 6752 (time 1771401458.91 total 80.06 last 80.44) - open/close 7582 (time 1771401468.91 total 90.06 last 82.99) - open/close 8398 (time 1771401478.92 total 100.07 last 81.53) - open/close 9199 (time 1771401488.92 total 110.07 last 80.08) total: 10000 open/close in 119.34 seconds: 83.79 ops/second - unlinked 0 (time 1771401502 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 676 (time 1771401586.40 total 10.01 last 67.56) - open/close 1426 (time 1771401596.40 total 20.01 last 74.94) - open/close 2235 (time 1771401606.42 total 30.03 last 80.74) - open/close 3052 (time 1771401616.43 total 40.04 last 81.67) - open/close 3958 (time 1771401626.43 total 50.04 last 90.56) - open/close 4777 (time 1771401636.44 total 60.05 last 81.83) - open/close 5559 (time 1771401646.45 total 70.05 last 78.16) - open/close 6427 (time 1771401656.45 total 80.06 last 86.74) - open/close 7182 (time 1771401666.46 total 90.07 last 75.45) - open/close 7962 (time 1771401676.46 total 100.07 last 77.98) - open/close 8803 (time 1771401686.46 total 110.07 last 84.07) - open/close 9638 (time 1771401696.47 total 120.08 last 83.48) total: 10000 open/close in 123.84 seconds: 80.75 ops/second - unlinked 0 (time 1771401704 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1014 (time 1771401789.83 total 10.01 last 101.34) - open/close 1846 (time 1771401799.85 total 20.02 last 83.04) - open/close 2712 (time 1771401809.86 total 30.03 last 86.55) - open/close 3633 (time 1771401819.86 total 40.03 last 92.07) - open/close 4543 (time 1771401829.87 total 50.04 last 90.94) - open/close 5431 (time 1771401839.88 total 60.05 last 88.72) - open/close 6301 (time 1771401849.88 total 70.05 last 86.96) - open/close 7111 (time 1771401859.90 total 80.07 last 80.90) - open/close 7918 (time 1771401869.91 total 90.08 last 80.60) - open/close 8687 (time 1771401879.92 total 100.09 last 76.79) - open/close 9501 (time 1771401889.93 total 110.10 last 81.37) total: 10000 open/close in 116.11 seconds: 86.13 ops/second - unlinked 0 (time 1771401900 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 809 (time 1771401980.52 total 10.01 last 80.81) - open/close 1696 (time 1771401990.52 total 20.02 last 88.65) - open/close 2605 (time 1771402000.53 total 30.02 last 90.86) - open/close 3504 (time 1771402010.53 total 40.02 last 89.88) - open/close 4372 (time 1771402020.53 total 50.02 last 86.80) - open/close 5273 (time 1771402030.54 total 60.03 last 90.02) - open/close 6205 (time 1771402040.54 total 70.03 last 93.18) - open/close 7093 (time 1771402050.55 total 80.04 last 88.74) - open/close 8022 (time 1771402060.56 total 90.05 last 92.84) - open/close 9014 (time 1771402070.56 total 100.05 last 99.18) - open/close 9910 (time 1771402080.56 total 110.06 last 89.56) total: 10000 open/close in 111.00 seconds: 90.09 ops/second - unlinked 0 (time 1771402085 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2