== conf-sanity test 69: replace an OST with the same index ========================================================== 09:58:22 (1761314302) start mds service on oleg212-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: detected 4 online CPUs by sysfs oleg212-server: Force libcfs to create 2 CPU partitions oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg212-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock oleg212-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 785 (time 1761314380.45 total 10.02 last 78.37) - open/close 1605 (time 1761314390.46 total 20.02 last 81.99) - open/close 2404 (time 1761314400.46 total 30.03 last 79.84) - open/close 3153 (time 1761314410.47 total 40.03 last 74.85) - open/close 3949 (time 1761314420.47 total 50.03 last 79.58) - open/close 4714 (time 1761314430.47 total 60.04 last 76.48) - open/close 5505 (time 1761314440.49 total 70.05 last 78.97) - open/close 6299 (time 1761314450.49 total 80.06 last 79.38) - open/close 7054 (time 1761314460.50 total 90.07 last 75.43) - open/close 7828 (time 1761314470.51 total 100.07 last 77.34) - open/close 8588 (time 1761314480.52 total 110.08 last 75.96) - open/close 9372 (time 1761314490.52 total 120.08 last 78.36) total: 10000 open/close in 127.84 seconds: 78.23 ops/second - unlinked 0 (time 1761314501 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 816 (time 1761314595.23 total 10.01 last 81.55) - open/close 1697 (time 1761314605.23 total 20.01 last 88.10) - open/close 2558 (time 1761314615.24 total 30.01 last 86.07) - open/close 3401 (time 1761314625.24 total 40.01 last 84.29) - open/close 4075 (time 1761314635.25 total 50.02 last 67.34) - open/close 4881 (time 1761314645.25 total 60.02 last 80.58) - open/close 5660 (time 1761314655.26 total 70.03 last 77.82) - open/close 6464 (time 1761314665.28 total 80.05 last 80.27) - open/close 7298 (time 1761314675.28 total 90.06 last 83.34) - open/close 8161 (time 1761314685.29 total 100.06 last 86.22) - open/close 8831 (time 1761314695.30 total 110.07 last 66.94) - open/close 9642 (time 1761314705.30 total 120.07 last 81.09) total: 10000 open/close in 124.59 seconds: 80.26 ops/second - unlinked 0 (time 1761314713 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 826 (time 1761314800.24 total 10.01 last 82.54) - open/close 1606 (time 1761314810.25 total 20.02 last 77.92) - open/close 2452 (time 1761314820.25 total 30.02 last 84.54) - open/close 3306 (time 1761314830.27 total 40.04 last 85.25) - open/close 4130 (time 1761314840.27 total 50.04 last 82.39) - open/close 4933 (time 1761314850.29 total 60.06 last 80.14) - open/close 5753 (time 1761314860.29 total 70.07 last 81.99) - open/close 6547 (time 1761314870.30 total 80.07 last 79.37) - open/close 7334 (time 1761314880.30 total 90.07 last 78.65) - open/close 8141 (time 1761314890.31 total 100.08 last 80.67) - open/close 8846 (time 1761314900.31 total 110.08 last 70.49) - open/close 9416 (time 1761314910.32 total 120.09 last 56.95) total: 10000 open/close in 127.51 seconds: 78.43 ops/second - unlinked 0 (time 1761314920 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 821 (time 1761315008.86 total 10.01 last 82.00) - open/close 1593 (time 1761315018.87 total 20.02 last 77.16) - open/close 2380 (time 1761315028.87 total 30.02 last 78.65) - open/close 3141 (time 1761315038.88 total 40.03 last 76.03) - open/close 3978 (time 1761315048.89 total 50.04 last 83.65) - open/close 4720 (time 1761315058.89 total 60.04 last 74.17) - open/close 5533 (time 1761315068.90 total 70.05 last 81.28) - open/close 6358 (time 1761315078.90 total 80.05 last 82.45) - open/close 7173 (time 1761315088.91 total 90.06 last 81.44) - open/close 7966 (time 1761315098.92 total 100.07 last 79.24) - open/close 8779 (time 1761315108.92 total 110.07 last 81.28) - open/close 9567 (time 1761315118.92 total 120.07 last 78.80) total: 10000 open/close in 125.39 seconds: 79.75 ops/second - unlinked 0 (time 1761315127 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 666 (time 1761315213.87 total 10.00 last 66.58) - open/close 1366 (time 1761315223.87 total 20.01 last 69.99) - open/close 2143 (time 1761315233.88 total 30.01 last 77.66) - open/close 2976 (time 1761315243.88 total 40.01 last 83.27) - open/close 3783 (time 1761315253.89 total 50.02 last 80.69) - open/close 4580 (time 1761315263.89 total 60.02 last 79.64) - open/close 5367 (time 1761315273.90 total 70.03 last 78.65) - open/close 6052 (time 1761315283.91 total 80.04 last 68.44) - open/close 6738 (time 1761315293.91 total 90.04 last 68.59) - open/close 7464 (time 1761315303.92 total 100.05 last 72.50) - open/close 8270 (time 1761315313.93 total 110.06 last 80.53) - open/close 9006 (time 1761315323.94 total 120.07 last 73.57) - open/close 9694 (time 1761315333.95 total 130.08 last 68.73) total: 10000 open/close in 134.32 seconds: 74.45 ops/second - unlinked 0 (time 1761315342 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 813 (time 1761315431.03 total 10.01 last 81.24) - open/close 1571 (time 1761315441.04 total 20.01 last 75.77) - open/close 2413 (time 1761315451.04 total 30.01 last 84.17) - open/close 3211 (time 1761315461.04 total 40.02 last 79.78) - open/close 3957 (time 1761315471.05 total 50.02 last 74.56) - open/close 4780 (time 1761315481.06 total 60.04 last 82.18) - open/close 5557 (time 1761315491.08 total 70.05 last 77.57) - open/close 6379 (time 1761315501.09 total 80.06 last 82.13) - open/close 7176 (time 1761315511.09 total 90.07 last 79.67) - open/close 7978 (time 1761315521.10 total 100.07 last 80.17) - open/close 8786 (time 1761315531.10 total 110.08 last 80.76) - open/close 9625 (time 1761315541.11 total 120.08 last 83.83) total: 10000 open/close in 124.38 seconds: 80.40 ops/second - unlinked 0 (time 1761315549 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 781 (time 1761315639.57 total 10.00 last 78.08) - open/close 1558 (time 1761315649.58 total 20.01 last 77.64) - open/close 2386 (time 1761315659.59 total 30.02 last 82.74) - open/close 3255 (time 1761315669.59 total 40.03 last 86.83) - open/close 4080 (time 1761315679.60 total 50.04 last 82.41) - open/close 4890 (time 1761315689.61 total 60.04 last 80.94) - open/close 5652 (time 1761315699.61 total 70.05 last 76.19) - open/close 6472 (time 1761315709.62 total 80.05 last 81.93) - open/close 7186 (time 1761315719.63 total 90.06 last 71.37) - open/close 7994 (time 1761315729.63 total 100.06 last 80.80) - open/close 8784 (time 1761315739.64 total 110.07 last 78.89) - open/close 9600 (time 1761315749.65 total 120.09 last 81.50) total: 10000 open/close in 124.91 seconds: 80.06 ops/second - unlinked 0 (time 1761315757 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 825 (time 1761315844.34 total 10.00 last 82.46) - open/close 1603 (time 1761315854.35 total 20.02 last 77.72) - open/close 2375 (time 1761315864.35 total 30.02 last 77.18) - open/close 3199 (time 1761315874.35 total 40.02 last 82.38) - open/close 3938 (time 1761315884.36 total 50.03 last 73.83) - open/close 4714 (time 1761315894.37 total 60.04 last 77.54) - open/close 5434 (time 1761315904.37 total 70.04 last 71.99) - open/close 6195 (time 1761315914.38 total 80.05 last 76.03) - open/close 6996 (time 1761315924.38 total 90.05 last 80.07) - open/close 7778 (time 1761315934.39 total 100.05 last 78.18) - open/close 8538 (time 1761315944.39 total 110.06 last 75.95) - open/close 9342 (time 1761315954.40 total 120.07 last 80.33) total: 10000 open/close in 128.22 seconds: 77.99 ops/second - unlinked 0 (time 1761315965 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 754 (time 1761316055.57 total 10.01 last 75.36) - open/close 1502 (time 1761316065.58 total 20.01 last 74.73) - open/close 2287 (time 1761316075.58 total 30.02 last 78.46) - open/close 3116 (time 1761316085.59 total 40.03 last 82.83) - open/close 3923 (time 1761316095.60 total 50.03 last 80.67) - open/close 4710 (time 1761316105.60 total 60.04 last 78.66) - open/close 5494 (time 1761316115.61 total 70.04 last 78.35) - open/close 6268 (time 1761316125.62 total 80.06 last 77.28) - open/close 6955 (time 1761316135.63 total 90.06 last 68.66) - open/close 7601 (time 1761316145.64 total 100.07 last 64.56) - open/close 8407 (time 1761316155.67 total 110.10 last 80.33) - open/close 9253 (time 1761316165.67 total 120.11 last 84.56) total: 10000 open/close in 129.94 seconds: 76.96 ops/second - unlinked 0 (time 1761316179 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 739 (time 1761316270.56 total 10.00 last 73.89) - open/close 1516 (time 1761316280.56 total 20.00 last 77.70) - open/close 2319 (time 1761316290.57 total 30.01 last 80.23) - open/close 3095 (time 1761316300.58 total 40.02 last 77.50) - open/close 3840 (time 1761316310.60 total 50.04 last 74.39) - open/close 4601 (time 1761316320.60 total 60.04 last 76.07) - open/close 5391 (time 1761316330.61 total 70.05 last 78.95) - open/close 6145 (time 1761316340.62 total 80.06 last 75.29) - open/close 6886 (time 1761316350.62 total 90.06 last 74.09) - open/close 7672 (time 1761316360.63 total 100.07 last 78.57) - open/close 8476 (time 1761316370.64 total 110.08 last 80.32) - open/close 9194 (time 1761316380.64 total 120.08 last 71.77) - open/close 9872 (time 1761316390.65 total 130.10 last 67.70) total: 10000 open/close in 131.98 seconds: 75.77 ops/second - unlinked 0 (time 1761316396 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.112@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.112@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg212-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg212-server: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg212-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock oleg212-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg212-server: LNET ready to unload oleg212-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2