== conf-sanity test 69: replace an OST with the same index ========================================================== 11:11:38 (1761318698) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 807 (time 1761318767.06 total 10.01 last 80.64) - open/close 1647 (time 1761318777.07 total 20.01 last 83.94) - open/close 2488 (time 1761318787.07 total 30.02 last 84.09) - open/close 3289 (time 1761318797.07 total 40.02 last 80.08) - open/close 4071 (time 1761318807.08 total 50.02 last 78.19) - open/close 4722 (time 1761318817.08 total 60.03 last 65.06) - open/close 5464 (time 1761318827.09 total 70.03 last 74.13) - open/close 6258 (time 1761318837.10 total 80.04 last 79.34) - open/close 7078 (time 1761318847.10 total 90.05 last 81.97) - open/close 7857 (time 1761318857.12 total 100.06 last 77.80) - open/close 8715 (time 1761318867.13 total 110.07 last 85.70) - open/close 9549 (time 1761318877.13 total 120.07 last 83.39) total: 10000 open/close in 125.60 seconds: 79.62 ops/second - unlinked 0 (time 1761318886 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 913 (time 1761318964.87 total 10.01 last 91.23) - open/close 1741 (time 1761318974.88 total 20.02 last 82.73) - open/close 2589 (time 1761318984.88 total 30.02 last 84.77) - open/close 3425 (time 1761318994.89 total 40.03 last 83.53) - open/close 4252 (time 1761319004.90 total 50.04 last 82.59) - open/close 5105 (time 1761319014.91 total 60.05 last 85.25) - open/close 6040 (time 1761319024.92 total 70.06 last 93.40) - open/close 7017 (time 1761319034.93 total 80.07 last 97.63) - open/close 7965 (time 1761319044.93 total 90.07 last 94.77) - open/close 8969 (time 1761319054.95 total 100.08 last 100.25) - open/close 9822 (time 1761319064.95 total 110.09 last 85.22) total: 10000 open/close in 112.27 seconds: 89.07 ops/second - unlinked 0 (time 1761319069 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1167 (time 1761319136.55 total 10.00 last 116.65) - open/close 2082 (time 1761319146.56 total 20.01 last 91.45) - open/close 2946 (time 1761319156.57 total 30.02 last 86.32) - open/close 3913 (time 1761319166.57 total 40.02 last 96.67) - open/close 4928 (time 1761319176.57 total 50.02 last 101.48) - open/close 6048 (time 1761319186.58 total 60.03 last 111.93) - open/close 7152 (time 1761319196.58 total 70.03 last 110.39) - open/close 8409 (time 1761319206.59 total 80.04 last 125.62) - open/close 9396 (time 1761319216.59 total 90.04 last 98.69) total: 10000 open/close in 96.94 seconds: 103.15 ops/second - unlinked 0 (time 1761319226 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1017 (time 1761319294.26 total 10.01 last 101.61) - open/close 2385 (time 1761319304.26 total 20.02 last 136.71) - open/close 3560 (time 1761319314.26 total 30.02 last 117.49) - open/close 4560 (time 1761319324.27 total 40.02 last 99.93) - open/close 5551 (time 1761319334.28 total 50.03 last 99.03) - open/close 6942 (time 1761319344.29 total 60.04 last 138.99) - open/close 8403 (time 1761319354.30 total 70.05 last 145.95) - open/close 9765 (time 1761319364.31 total 80.06 last 136.06) total: 10000 open/close in 82.36 seconds: 121.42 ops/second - unlinked 0 (time 1761319369 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1188 (time 1761319428.90 total 10.00 last 118.76) - open/close 2473 (time 1761319438.91 total 20.01 last 128.46) - open/close 3675 (time 1761319448.91 total 30.01 last 120.19) - open/close 4997 (time 1761319458.91 total 40.01 last 132.20) - open/close 6651 (time 1761319468.91 total 50.01 last 165.35) - open/close 8040 (time 1761319478.91 total 60.01 last 138.85) - open/close 9396 (time 1761319488.92 total 70.02 last 135.53) total: 10000 open/close in 74.04 seconds: 135.06 ops/second - unlinked 0 (time 1761319494 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1500 (time 1761319553.21 total 10.01 last 149.92) - open/close 2738 (time 1761319563.21 total 20.01 last 123.78) - open/close 3780 (time 1761319573.22 total 30.01 last 104.15) - open/close 4904 (time 1761319583.22 total 40.02 last 112.32) - open/close 6098 (time 1761319593.24 total 50.04 last 119.15) - open/close 6707 (time 1761319603.25 total 60.04 last 60.89) - open/close 7278 (time 1761319613.25 total 70.05 last 57.08) - open/close 7816 (time 1761319623.26 total 80.05 last 53.76) - open/close 8324 (time 1761319633.27 total 90.07 last 50.74) - open/close 8910 (time 1761319643.28 total 100.08 last 58.55) - open/close 9592 (time 1761319653.28 total 110.08 last 68.17) total: 10000 open/close in 115.29 seconds: 86.74 ops/second - unlinked 0 (time 1761319661 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 825 (time 1761319748.42 total 10.00 last 82.47) - open/close 1611 (time 1761319758.43 total 20.01 last 78.53) - open/close 2427 (time 1761319768.43 total 30.02 last 81.56) - open/close 3235 (time 1761319778.43 total 40.02 last 80.80) - open/close 4042 (time 1761319788.45 total 50.03 last 80.59) - open/close 4906 (time 1761319798.45 total 60.03 last 86.40) - open/close 5695 (time 1761319808.45 total 70.03 last 78.88) - open/close 6551 (time 1761319818.45 total 80.04 last 85.58) - open/close 7307 (time 1761319828.46 total 90.04 last 75.54) - open/close 8022 (time 1761319838.47 total 100.05 last 71.45) - open/close 8823 (time 1761319848.48 total 110.06 last 80.02) - open/close 9636 (time 1761319858.49 total 120.07 last 81.21) total: 10000 open/close in 124.84 seconds: 80.10 ops/second - unlinked 0 (time 1761319866 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 753 (time 1761319957.57 total 10.01 last 75.23) - open/close 1537 (time 1761319967.58 total 20.02 last 78.34) - open/close 2378 (time 1761319977.58 total 30.02 last 84.07) - open/close 3183 (time 1761319987.60 total 40.04 last 80.38) - open/close 3952 (time 1761319997.60 total 50.04 last 76.88) - open/close 4790 (time 1761320007.60 total 60.04 last 83.78) - open/close 5540 (time 1761320017.61 total 70.05 last 74.95) - open/close 6293 (time 1761320027.61 total 80.06 last 75.25) - open/close 7130 (time 1761320037.62 total 90.06 last 83.66) - open/close 7927 (time 1761320047.62 total 100.07 last 79.66) - open/close 8671 (time 1761320057.63 total 110.07 last 74.37) - open/close 9447 (time 1761320067.63 total 120.08 last 77.54) total: 10000 open/close in 127.22 seconds: 78.61 ops/second - unlinked 0 (time 1761320078 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 783 (time 1761320162.89 total 10.01 last 78.23) - open/close 1576 (time 1761320172.90 total 20.02 last 79.22) - open/close 2362 (time 1761320182.91 total 30.03 last 78.51) - open/close 3147 (time 1761320192.92 total 40.03 last 78.48) - open/close 3971 (time 1761320202.92 total 50.04 last 82.38) - open/close 4772 (time 1761320212.92 total 60.04 last 80.07) - open/close 5537 (time 1761320222.94 total 70.06 last 76.35) - open/close 6342 (time 1761320232.95 total 80.07 last 80.45) - open/close 7139 (time 1761320242.95 total 90.07 last 79.67) - open/close 7933 (time 1761320252.96 total 100.08 last 79.33) - open/close 8746 (time 1761320262.98 total 110.09 last 81.16) - open/close 9539 (time 1761320272.98 total 120.10 last 79.27) total: 10000 open/close in 125.47 seconds: 79.70 ops/second - unlinked 0 (time 1761320281 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 856 (time 1761320357.66 total 10.00 last 85.56) - open/close 1693 (time 1761320367.66 total 20.01 last 83.66) - open/close 2644 (time 1761320377.67 total 30.02 last 94.98) - open/close 3509 (time 1761320387.67 total 40.02 last 86.49) - open/close 4368 (time 1761320397.69 total 50.03 last 85.81) - open/close 5238 (time 1761320407.69 total 60.04 last 86.92) - open/close 6152 (time 1761320417.70 total 70.04 last 91.39) - open/close 6984 (time 1761320427.70 total 80.05 last 83.17) - open/close 7733 (time 1761320437.71 total 90.06 last 74.84) - open/close 8550 (time 1761320447.71 total 100.06 last 81.66) - open/close 9360 (time 1761320457.72 total 110.07 last 80.93) total: 10000 open/close in 116.78 seconds: 85.63 ops/second - unlinked 0 (time 1761320467 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre On OST0, 2701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg114-server: LNET ready to unload oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2