== conf-sanity test 69: replace an OST with the same index ========================================================== 13:18:35 (1761239915) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 629 (time 1761239984.83 total 10.00 last 62.88) - open/close 1297 (time 1761239994.83 total 20.01 last 66.76) - open/close 2018 (time 1761240004.83 total 30.01 last 72.09) - open/close 2804 (time 1761240014.84 total 40.02 last 78.52) - open/close 3523 (time 1761240024.85 total 50.03 last 71.84) - open/close 4263 (time 1761240034.85 total 60.03 last 73.99) - open/close 4984 (time 1761240044.86 total 70.04 last 72.06) - open/close 5703 (time 1761240054.86 total 80.04 last 71.87) - open/close 6402 (time 1761240064.88 total 90.05 last 69.80) - open/close 7111 (time 1761240074.88 total 100.05 last 70.90) - open/close 7835 (time 1761240084.89 total 110.06 last 72.32) - open/close 8559 (time 1761240094.90 total 120.07 last 72.33) - open/close 9263 (time 1761240104.91 total 130.08 last 70.33) total: 10000 open/close in 139.88 seconds: 71.49 ops/second - unlinked 0 (time 1761240119 ; total 1 ; last 1) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 758 (time 1761240217.34 total 10.01 last 75.73) - open/close 1475 (time 1761240227.35 total 20.01 last 71.69) - open/close 2305 (time 1761240237.35 total 30.01 last 82.97) - open/close 3264 (time 1761240247.35 total 40.02 last 95.88) - open/close 4091 (time 1761240257.36 total 50.03 last 82.59) - open/close 4915 (time 1761240267.36 total 60.03 last 82.40) - open/close 5740 (time 1761240277.38 total 70.05 last 82.36) - open/close 6494 (time 1761240287.39 total 80.05 last 75.36) - open/close 7250 (time 1761240297.39 total 90.05 last 75.60) - open/close 8012 (time 1761240307.39 total 100.06 last 76.16) - open/close 8723 (time 1761240317.40 total 110.07 last 71.04) - open/close 9486 (time 1761240327.41 total 120.07 last 76.25) total: 10000 open/close in 126.27 seconds: 79.19 ops/second - unlinked 0 (time 1761240337 ; total 1 ; last 1) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 934 (time 1761240417.49 total 10.00 last 93.37) - open/close 1788 (time 1761240427.50 total 20.01 last 85.33) - open/close 2683 (time 1761240437.50 total 30.02 last 89.44) - open/close 3560 (time 1761240447.51 total 40.03 last 87.62) - open/close 4579 (time 1761240457.52 total 50.04 last 101.81) - open/close 5412 (time 1761240467.53 total 60.04 last 83.24) - open/close 6269 (time 1761240477.53 total 70.05 last 85.64) - open/close 7107 (time 1761240487.54 total 80.06 last 83.74) - open/close 8041 (time 1761240497.55 total 90.06 last 93.34) - open/close 8839 (time 1761240507.56 total 100.07 last 79.73) - open/close 9694 (time 1761240517.56 total 110.08 last 85.47) total: 10000 open/close in 113.74 seconds: 87.92 ops/second - unlinked 0 (time 1761240524 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 748 (time 1761240609.82 total 10.00 last 74.77) - open/close 1552 (time 1761240619.83 total 20.01 last 80.32) - open/close 2351 (time 1761240629.83 total 30.01 last 79.90) - open/close 3155 (time 1761240639.83 total 40.01 last 80.40) - open/close 3937 (time 1761240649.83 total 50.02 last 78.19) - open/close 4703 (time 1761240659.85 total 60.03 last 76.47) - open/close 5494 (time 1761240669.85 total 70.03 last 79.08) - open/close 6242 (time 1761240679.85 total 80.04 last 74.76) - open/close 7035 (time 1761240689.86 total 90.05 last 79.24) - open/close 7841 (time 1761240699.86 total 100.05 last 80.58) - open/close 8677 (time 1761240709.87 total 110.06 last 83.52) - open/close 9677 (time 1761240719.87 total 120.06 last 99.99) total: 10000 open/close in 123.66 seconds: 80.87 ops/second - unlinked 0 (time 1761240726 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 765 (time 1761240805.50 total 10.00 last 76.49) - open/close 1548 (time 1761240815.51 total 20.01 last 78.20) - open/close 2402 (time 1761240825.52 total 30.02 last 85.33) - open/close 3246 (time 1761240835.52 total 40.02 last 84.40) - open/close 4055 (time 1761240845.53 total 50.03 last 80.86) - open/close 4881 (time 1761240855.53 total 60.04 last 82.54) - open/close 5692 (time 1761240865.53 total 70.04 last 81.09) - open/close 6539 (time 1761240875.54 total 80.04 last 84.68) - open/close 7313 (time 1761240885.54 total 90.05 last 77.35) - open/close 8087 (time 1761240895.56 total 100.07 last 77.25) - open/close 8953 (time 1761240905.57 total 110.08 last 86.51) - open/close 9872 (time 1761240915.58 total 120.09 last 91.82) total: 10000 open/close in 121.80 seconds: 82.10 ops/second - unlinked 0 (time 1761240920 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 923 (time 1761241005.81 total 10.01 last 92.21) - open/close 1766 (time 1761241015.81 total 20.02 last 84.24) - open/close 2564 (time 1761241025.83 total 30.03 last 79.70) - open/close 3446 (time 1761241035.83 total 40.04 last 88.14) - open/close 4181 (time 1761241045.85 total 50.05 last 73.38) - open/close 4988 (time 1761241055.85 total 60.05 last 80.69) - open/close 5788 (time 1761241065.86 total 70.07 last 79.89) - open/close 6548 (time 1761241075.87 total 80.07 last 75.94) - open/close 7303 (time 1761241085.88 total 90.08 last 75.44) - open/close 8052 (time 1761241095.89 total 100.09 last 74.85) - open/close 8761 (time 1761241105.91 total 110.11 last 70.75) - open/close 9426 (time 1761241115.93 total 120.14 last 66.34) total: 10000 open/close in 128.32 seconds: 77.93 ops/second - unlinked 0 (time 1761241127 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 796 (time 1761241217.30 total 10.01 last 79.50) - open/close 1601 (time 1761241227.31 total 20.02 last 80.42) - open/close 2441 (time 1761241237.32 total 30.03 last 83.95) - open/close 3239 (time 1761241247.33 total 40.04 last 79.72) - open/close 3996 (time 1761241257.33 total 50.04 last 75.66) - open/close 4761 (time 1761241267.33 total 60.05 last 76.48) - open/close 5473 (time 1761241277.35 total 70.06 last 71.12) - open/close 6299 (time 1761241287.35 total 80.06 last 82.57) - open/close 7096 (time 1761241297.36 total 90.07 last 79.66) - open/close 7941 (time 1761241307.36 total 100.07 last 84.44) - open/close 8639 (time 1761241317.36 total 110.08 last 69.79) - open/close 9392 (time 1761241327.38 total 120.09 last 75.20) total: 10000 open/close in 128.45 seconds: 77.85 ops/second - unlinked 0 (time 1761241339 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 724 (time 1761241426.40 total 10.00 last 72.39) - open/close 1453 (time 1761241436.42 total 20.02 last 72.80) - open/close 2272 (time 1761241446.42 total 30.02 last 81.84) - open/close 3073 (time 1761241456.43 total 40.03 last 80.05) - open/close 3844 (time 1761241466.44 total 50.04 last 77.04) - open/close 4653 (time 1761241476.45 total 60.04 last 80.84) - open/close 5414 (time 1761241486.45 total 70.05 last 76.03) - open/close 6122 (time 1761241496.46 total 80.06 last 70.78) - open/close 6865 (time 1761241506.46 total 90.06 last 74.25) - open/close 7671 (time 1761241516.47 total 100.07 last 80.57) - open/close 8490 (time 1761241526.48 total 110.08 last 81.82) - open/close 9260 (time 1761241536.48 total 120.08 last 76.97) - open/close 9989 (time 1761241546.48 total 130.08 last 72.89) total: 10000 open/close in 130.25 seconds: 76.78 ops/second - unlinked 0 (time 1761241550 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 915 (time 1761241636.41 total 10.00 last 91.48) - open/close 1786 (time 1761241646.42 total 20.01 last 87.03) - open/close 2602 (time 1761241656.43 total 30.02 last 81.52) - open/close 3400 (time 1761241666.44 total 40.03 last 79.73) - open/close 4230 (time 1761241676.45 total 50.04 last 82.92) - open/close 5019 (time 1761241686.46 total 60.04 last 78.85) - open/close 5758 (time 1761241696.47 total 70.05 last 73.82) - open/close 6530 (time 1761241706.48 total 80.07 last 77.09) - open/close 7386 (time 1761241716.49 total 90.08 last 85.53) - open/close 8194 (time 1761241726.50 total 100.08 last 80.75) - open/close 8987 (time 1761241736.50 total 110.09 last 79.25) - open/close 9800 (time 1761241746.50 total 120.09 last 81.29) total: 10000 open/close in 122.39 seconds: 81.71 ops/second - unlinked 0 (time 1761241751 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 847 (time 1761241837.17 total 10.01 last 84.60) - open/close 1721 (time 1761241847.18 total 20.02 last 87.35) - open/close 2552 (time 1761241857.18 total 30.02 last 83.08) - open/close 3425 (time 1761241867.20 total 40.04 last 87.14) - open/close 4233 (time 1761241877.21 total 50.05 last 80.68) - open/close 5006 (time 1761241887.22 total 60.06 last 77.25) - open/close 5648 (time 1761241897.22 total 70.06 last 64.18) - open/close 6435 (time 1761241907.22 total 80.07 last 78.68) - open/close 7174 (time 1761241917.23 total 90.07 last 73.89) - open/close 7939 (time 1761241927.23 total 100.08 last 76.43) - open/close 8670 (time 1761241937.25 total 110.09 last 72.98) - open/close 9383 (time 1761241947.26 total 120.10 last 71.24) total: 10000 open/close in 127.61 seconds: 78.37 ops/second - unlinked 0 (time 1761241957 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg346-server: LNET ready to unload oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2