== conf-sanity test 69: replace an OST with the same index ========================================================== 12:52:19 (1786812739) start mds service on oleg217-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg217-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg217-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 858 (time 1786812858.62 total 10.01 last 85.74) - open/close 1638 (time 1786812868.63 total 20.01 last 77.98) - open/close 2423 (time 1786812878.63 total 30.02 last 78.46) - open/close 3093 (time 1786812888.63 total 40.02 last 66.99) - open/close 3828 (time 1786812898.63 total 50.02 last 73.49) - open/close 4500 (time 1786812908.65 total 60.03 last 67.10) - open/close 5364 (time 1786812918.65 total 70.04 last 86.37) - open/close 6409 (time 1786812928.66 total 80.04 last 104.45) - open/close 7074 (time 1786812938.66 total 90.05 last 66.47) - open/close 7873 (time 1786812948.67 total 100.05 last 79.83) - open/close 8532 (time 1786812958.67 total 110.06 last 65.88) - open/close 9177 (time 1786812968.67 total 120.06 last 64.50) - open/close 9922 (time 1786812978.68 total 130.06 last 74.47) total: 10000 open/close in 130.87 seconds: 76.41 ops/second - unlinked 0 (time 1786812984 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 722 (time 1786813079.22 total 10.01 last 72.11) - open/close 1624 (time 1786813089.22 total 20.02 last 90.15) - open/close 2310 (time 1786813099.25 total 30.05 last 68.41) - open/close 3308 (time 1786813109.26 total 40.05 last 99.77) - open/close 4251 (time 1786813119.26 total 50.05 last 94.27) - open/close 5091 (time 1786813129.26 total 60.05 last 83.98) - open/close 6071 (time 1786813139.26 total 70.06 last 97.98) - open/close 6899 (time 1786813149.27 total 80.06 last 82.74) - open/close 7739 (time 1786813159.28 total 90.07 last 83.93) - open/close 8603 (time 1786813169.28 total 100.07 last 86.38) - open/close 9387 (time 1786813179.28 total 110.08 last 78.36) total: 10000 open/close in 117.44 seconds: 85.15 ops/second - unlinked 0 (time 1786813190 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 853 (time 1786813276.95 total 10.00 last 85.29) - open/close 1552 (time 1786813286.95 total 20.00 last 69.89) - open/close 2384 (time 1786813296.95 total 30.01 last 83.14) - open/close 3210 (time 1786813306.96 total 40.02 last 82.54) - open/close 4169 (time 1786813316.96 total 50.02 last 95.89) - open/close 4998 (time 1786813326.97 total 60.02 last 82.87) - open/close 6023 (time 1786813336.97 total 70.02 last 102.49) - open/close 6897 (time 1786813346.98 total 80.03 last 87.32) - open/close 7618 (time 1786813356.98 total 90.04 last 72.07) - open/close 8299 (time 1786813366.99 total 100.05 last 68.03) - open/close 8751 (time 1786813377.02 total 110.08 last 45.05) - open/close 9358 (time 1786813387.03 total 120.08 last 60.68) total: 10000 open/close in 129.94 seconds: 76.96 ops/second - unlinked 0 (time 1786813401 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1081 (time 1786813482.42 total 10.01 last 108.00) - open/close 1957 (time 1786813492.43 total 20.01 last 87.57) - open/close 2903 (time 1786813502.43 total 30.02 last 94.58) - open/close 3693 (time 1786813512.43 total 40.02 last 78.97) - open/close 4642 (time 1786813522.43 total 50.02 last 94.87) - open/close 5632 (time 1786813532.44 total 60.03 last 98.91) - open/close 6646 (time 1786813542.48 total 70.06 last 101.07) - open/close 7515 (time 1786813552.49 total 80.07 last 86.81) - open/close 8237 (time 1786813562.51 total 90.10 last 72.04) - open/close 8902 (time 1786813572.51 total 100.10 last 66.49) - open/close 9796 (time 1786813582.52 total 110.10 last 89.35) total: 10000 open/close in 112.12 seconds: 89.19 ops/second - unlinked 0 (time 1786813589 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1000 (time 1786813667.64 total 10.01 last 99.89) - open/close 1908 (time 1786813677.64 total 20.01 last 90.77) - open/close 2793 (time 1786813687.65 total 30.02 last 88.48) - open/close 3818 (time 1786813697.66 total 40.03 last 102.39) - open/close 4810 (time 1786813707.69 total 50.06 last 98.85) - open/close 5825 (time 1786813717.70 total 60.07 last 101.40) - open/close 6816 (time 1786813727.71 total 70.08 last 99.05) - open/close 7625 (time 1786813737.72 total 80.09 last 80.82) - open/close 8487 (time 1786813747.72 total 90.09 last 86.15) - open/close 9305 (time 1786813757.73 total 100.10 last 81.73) total: 10000 open/close in 109.85 seconds: 91.03 ops/second - unlinked 0 (time 1786813774 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1070 (time 1786813849.71 total 10.00 last 106.98) - open/close 2146 (time 1786813859.72 total 20.01 last 107.52) - open/close 3234 (time 1786813869.72 total 30.01 last 108.74) - open/close 4267 (time 1786813879.74 total 40.03 last 103.17) - open/close 5345 (time 1786813889.74 total 50.03 last 107.78) - open/close 6179 (time 1786813899.75 total 60.04 last 83.34) - open/close 7203 (time 1786813909.75 total 70.04 last 102.36) - open/close 8129 (time 1786813919.76 total 80.05 last 92.52) - open/close 8861 (time 1786813929.76 total 90.05 last 73.18) - open/close 9786 (time 1786813939.76 total 100.05 last 92.47) total: 10000 open/close in 102.76 seconds: 97.31 ops/second - unlinked 0 (time 1786813946 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 903 (time 1786814017.52 total 10.01 last 90.20) - open/close 1819 (time 1786814027.53 total 20.02 last 91.52) - open/close 2726 (time 1786814037.53 total 30.02 last 90.68) - open/close 3694 (time 1786814047.53 total 40.02 last 96.78) - open/close 4698 (time 1786814057.54 total 50.03 last 100.30) - open/close 5698 (time 1786814067.55 total 60.04 last 99.98) - open/close 6540 (time 1786814077.55 total 70.04 last 84.19) - open/close 7667 (time 1786814087.55 total 80.04 last 112.70) - open/close 8698 (time 1786814097.56 total 90.05 last 103.00) - open/close 9696 (time 1786814107.56 total 100.05 last 99.78) total: 10000 open/close in 102.92 seconds: 97.16 ops/second - unlinked 0 (time 1786814115 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 795 (time 1786814193.58 total 10.01 last 79.41) - open/close 1684 (time 1786814203.59 total 20.02 last 88.84) - open/close 2677 (time 1786814213.59 total 30.02 last 99.25) - open/close 3512 (time 1786814223.60 total 40.03 last 83.47) - open/close 4477 (time 1786814233.60 total 50.03 last 96.45) - open/close 5490 (time 1786814243.61 total 60.04 last 101.21) - open/close 6445 (time 1786814253.61 total 70.04 last 95.46) - open/close 7387 (time 1786814263.61 total 80.04 last 94.20) - open/close 8351 (time 1786814273.62 total 90.05 last 96.37) - open/close 9307 (time 1786814283.62 total 100.05 last 95.57) total: 10000 open/close in 106.99 seconds: 93.46 ops/second - unlinked 0 (time 1786814294 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 658 (time 1786814375.84 total 10.00 last 65.77) - open/close 1271 (time 1786814385.85 total 20.01 last 61.24) - open/close 2234 (time 1786814395.85 total 30.02 last 96.29) - open/close 3361 (time 1786814405.86 total 40.03 last 112.60) - open/close 4214 (time 1786814415.86 total 50.03 last 85.27) - open/close 5207 (time 1786814425.86 total 60.03 last 99.29) - open/close 6303 (time 1786814435.87 total 70.04 last 109.55) - open/close 7343 (time 1786814445.87 total 80.04 last 103.94) - open/close 8358 (time 1786814455.88 total 90.05 last 101.40) - open/close 9328 (time 1786814465.90 total 100.07 last 96.86) total: 10000 open/close in 106.90 seconds: 93.54 ops/second - unlinked 0 (time 1786814476 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 843 (time 1786814547.31 total 10.01 last 84.20) - open/close 1731 (time 1786814557.32 total 20.02 last 88.71) - open/close 2523 (time 1786814567.33 total 30.03 last 79.16) - open/close 3404 (time 1786814577.33 total 40.03 last 88.09) - open/close 4373 (time 1786814587.33 total 50.03 last 96.86) - open/close 5332 (time 1786814597.36 total 60.06 last 95.67) - open/close 6301 (time 1786814607.37 total 70.07 last 96.78) - open/close 7267 (time 1786814617.37 total 80.07 last 96.59) - open/close 8433 (time 1786814627.38 total 90.08 last 116.53) - open/close 9496 (time 1786814637.39 total 100.08 last 106.22) total: 10000 open/close in 105.21 seconds: 95.04 ops/second - unlinked 0 (time 1786814646 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.117@tcp sys.timeout=20 start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg217-server unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local oleg217-server: modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2