== conf-sanity test 69: replace an OST with the same index ========================================================== 10:02:29 (1761228149) start mds service on oleg418-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg418-server: detected 4 online CPUs by sysfs oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 771 (time 1761228228.83 total 10.01 last 77.01) - open/close 1593 (time 1761228238.84 total 20.02 last 82.17) - open/close 2587 (time 1761228248.84 total 30.02 last 99.37) - open/close 3467 (time 1761228258.85 total 40.03 last 87.94) - open/close 4294 (time 1761228268.85 total 50.03 last 82.67) - open/close 5040 (time 1761228278.86 total 60.04 last 74.52) - open/close 5778 (time 1761228288.87 total 70.05 last 73.71) - open/close 6497 (time 1761228298.88 total 80.06 last 71.84) - open/close 7347 (time 1761228308.90 total 90.07 last 84.89) - open/close 8149 (time 1761228318.91 total 100.08 last 80.13) - open/close 9016 (time 1761228328.91 total 110.09 last 86.66) - open/close 9871 (time 1761228338.92 total 120.10 last 85.43) total: 10000 open/close in 121.46 seconds: 82.33 ops/second - unlinked 0 (time 1761228343 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 859 (time 1761228425.83 total 10.00 last 85.86) - open/close 1710 (time 1761228435.84 total 20.01 last 85.03) - open/close 2529 (time 1761228445.84 total 30.02 last 81.87) - open/close 3329 (time 1761228455.85 total 40.03 last 79.93) - open/close 4162 (time 1761228465.86 total 50.03 last 83.23) - open/close 5067 (time 1761228475.86 total 60.03 last 90.49) - open/close 5962 (time 1761228485.87 total 70.04 last 89.42) - open/close 6758 (time 1761228495.87 total 80.05 last 79.58) - open/close 7587 (time 1761228505.88 total 90.05 last 82.87) - open/close 8414 (time 1761228515.88 total 100.06 last 82.65) - open/close 9256 (time 1761228525.88 total 110.06 last 84.19) total: 10000 open/close in 119.14 seconds: 83.93 ops/second - unlinked 0 (time 1761228537 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 866 (time 1761228620.39 total 10.02 last 86.41) - open/close 1746 (time 1761228630.40 total 20.03 last 87.93) - open/close 2559 (time 1761228640.40 total 30.03 last 81.30) - open/close 3473 (time 1761228650.41 total 40.04 last 91.35) - open/close 4389 (time 1761228660.42 total 50.05 last 91.51) - open/close 5292 (time 1761228670.43 total 60.05 last 90.22) - open/close 6103 (time 1761228680.43 total 70.05 last 81.09) - open/close 7088 (time 1761228690.43 total 80.06 last 98.47) - open/close 7928 (time 1761228700.44 total 90.06 last 83.95) - open/close 8765 (time 1761228710.44 total 100.07 last 83.67) - open/close 9589 (time 1761228720.45 total 110.08 last 82.33) total: 10000 open/close in 114.80 seconds: 87.11 ops/second - unlinked 0 (time 1761228728 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 743 (time 1761228811.26 total 10.01 last 74.22) - open/close 1525 (time 1761228821.26 total 20.01 last 78.19) - open/close 2415 (time 1761228831.27 total 30.02 last 88.93) - open/close 3273 (time 1761228841.27 total 40.02 last 85.79) - open/close 4064 (time 1761228851.27 total 50.02 last 79.07) - open/close 4885 (time 1761228861.28 total 60.03 last 82.04) - open/close 5679 (time 1761228871.30 total 70.05 last 79.23) - open/close 6351 (time 1761228881.30 total 80.06 last 67.19) - open/close 7138 (time 1761228891.31 total 90.06 last 78.67) - open/close 7943 (time 1761228901.31 total 100.07 last 80.44) - open/close 8653 (time 1761228911.31 total 110.07 last 70.99) - open/close 9511 (time 1761228921.31 total 120.07 last 85.80) total: 10000 open/close in 125.85 seconds: 79.46 ops/second - unlinked 0 (time 1761228929 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 832 (time 1761229012.55 total 10.01 last 83.12) - open/close 1685 (time 1761229022.56 total 20.01 last 85.27) - open/close 2520 (time 1761229032.56 total 30.02 last 83.46) - open/close 3334 (time 1761229042.57 total 40.02 last 81.37) - open/close 4131 (time 1761229052.57 total 50.02 last 79.69) - open/close 4940 (time 1761229062.57 total 60.02 last 80.90) - open/close 5696 (time 1761229072.58 total 70.03 last 75.52) - open/close 6513 (time 1761229082.59 total 80.04 last 81.62) - open/close 7343 (time 1761229092.59 total 90.05 last 82.94) - open/close 8411 (time 1761229102.60 total 100.06 last 106.73) - open/close 9344 (time 1761229112.60 total 110.06 last 93.25) total: 10000 open/close in 117.38 seconds: 85.20 ops/second - unlinked 0 (time 1761229122 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 853 (time 1761229199.15 total 10.00 last 85.29) - open/close 1563 (time 1761229209.15 total 20.01 last 70.94) - open/close 2349 (time 1761229219.16 total 30.01 last 78.58) - open/close 3215 (time 1761229229.16 total 40.02 last 86.55) - open/close 4048 (time 1761229239.17 total 50.02 last 83.25) - open/close 4850 (time 1761229249.17 total 60.03 last 80.16) - open/close 5663 (time 1761229259.19 total 70.04 last 81.20) - open/close 6532 (time 1761229269.20 total 80.05 last 86.81) - open/close 7419 (time 1761229279.20 total 90.05 last 88.68) - open/close 8312 (time 1761229289.21 total 100.06 last 89.23) - open/close 9202 (time 1761229299.21 total 110.06 last 88.99) total: 10000 open/close in 119.23 seconds: 83.87 ops/second - unlinked 0 (time 1761229312 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 876 (time 1761229392.94 total 10.00 last 87.56) - open/close 1756 (time 1761229402.95 total 20.02 last 87.90) - open/close 2620 (time 1761229412.96 total 30.02 last 86.33) - open/close 3429 (time 1761229422.96 total 40.03 last 80.86) - open/close 4229 (time 1761229432.98 total 50.05 last 79.86) - open/close 4979 (time 1761229442.98 total 60.05 last 74.98) - open/close 5751 (time 1761229452.99 total 70.05 last 77.17) - open/close 6522 (time 1761229462.99 total 80.06 last 77.07) - open/close 7301 (time 1761229472.99 total 90.06 last 77.86) - open/close 7978 (time 1761229483.00 total 100.07 last 67.67) - open/close 8661 (time 1761229493.00 total 110.07 last 68.28) - open/close 9436 (time 1761229503.02 total 120.08 last 77.37) total: 10000 open/close in 127.89 seconds: 78.19 ops/second - unlinked 0 (time 1761229513 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 852 (time 1761229599.48 total 10.01 last 85.15) - open/close 1655 (time 1761229609.48 total 20.01 last 80.25) - open/close 2500 (time 1761229619.49 total 30.01 last 84.47) - open/close 3260 (time 1761229629.50 total 40.03 last 75.92) - open/close 4011 (time 1761229639.51 total 50.03 last 75.04) - open/close 4744 (time 1761229649.51 total 60.04 last 73.26) - open/close 5568 (time 1761229659.52 total 70.05 last 82.30) - open/close 6375 (time 1761229669.52 total 80.05 last 80.68) - open/close 7220 (time 1761229679.53 total 90.06 last 84.43) - open/close 8064 (time 1761229689.55 total 100.08 last 84.28) - open/close 8912 (time 1761229699.55 total 110.08 last 84.80) - open/close 9709 (time 1761229709.56 total 120.08 last 79.63) total: 10000 open/close in 124.03 seconds: 80.63 ops/second - unlinked 0 (time 1761229716 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 703 (time 1761229798.55 total 10.02 last 70.19) - open/close 1550 (time 1761229808.56 total 20.02 last 84.68) - open/close 2404 (time 1761229818.56 total 30.03 last 85.34) - open/close 3250 (time 1761229828.57 total 40.03 last 84.53) - open/close 3955 (time 1761229838.57 total 50.03 last 70.50) - open/close 4737 (time 1761229848.57 total 60.04 last 78.17) - open/close 5515 (time 1761229858.58 total 70.05 last 77.74) - open/close 6216 (time 1761229868.59 total 80.06 last 70.01) - open/close 6974 (time 1761229878.59 total 90.06 last 75.80) - open/close 7813 (time 1761229888.60 total 100.06 last 83.86) - open/close 8548 (time 1761229898.60 total 110.07 last 73.48) - open/close 9283 (time 1761229908.61 total 120.07 last 73.44) total: 10000 open/close in 128.50 seconds: 77.82 ops/second - unlinked 0 (time 1761229920 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 748 (time 1761230012.28 total 10.01 last 74.76) - open/close 1600 (time 1761230022.28 total 20.01 last 85.20) - open/close 2397 (time 1761230032.28 total 30.01 last 79.66) - open/close 3203 (time 1761230042.29 total 40.02 last 80.51) - open/close 3863 (time 1761230052.30 total 50.03 last 65.95) - open/close 4669 (time 1761230062.31 total 60.04 last 80.52) - open/close 5542 (time 1761230072.32 total 70.05 last 87.25) - open/close 6389 (time 1761230082.32 total 80.05 last 84.67) - open/close 7173 (time 1761230092.34 total 90.06 last 78.29) - open/close 8027 (time 1761230102.35 total 100.08 last 85.27) - open/close 8864 (time 1761230112.35 total 110.08 last 83.68) - open/close 9606 (time 1761230122.36 total 120.09 last 74.17) total: 10000 open/close in 124.69 seconds: 80.20 ops/second - unlinked 0 (time 1761230130 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg418-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre On OST0, 4719 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg418-server: LNET ready to unload oleg418-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2