== conf-sanity test 69: replace an OST with the same index ========================================================== 10:30:20 (1761316220) start mds service on oleg337-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg337-server: detected 4 online CPUs by sysfs oleg337-server: Force libcfs to create 2 CPU partitions oleg337-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg337-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg337-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 904 (time 1761316282.33 total 10.01 last 90.31) - open/close 1758 (time 1761316292.34 total 20.02 last 85.34) - open/close 2811 (time 1761316302.34 total 30.02 last 105.25) - open/close 3664 (time 1761316312.34 total 40.02 last 85.27) - open/close 4386 (time 1761316322.35 total 50.03 last 72.19) - open/close 5201 (time 1761316332.35 total 60.03 last 81.48) - open/close 5981 (time 1761316342.35 total 70.03 last 77.98) - open/close 6806 (time 1761316352.35 total 80.03 last 82.49) - open/close 7566 (time 1761316362.35 total 90.03 last 75.98) - open/close 8312 (time 1761316372.36 total 100.04 last 74.59) - open/close 9023 (time 1761316382.36 total 110.04 last 71.05) - open/close 9843 (time 1761316392.38 total 120.06 last 81.90) total: 10000 open/close in 122.21 seconds: 81.82 ops/second - unlinked 0 (time 1761316397 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 904 (time 1761316478.60 total 10.00 last 90.36) - open/close 1642 (time 1761316488.61 total 20.01 last 73.74) - open/close 2372 (time 1761316498.62 total 30.02 last 72.96) - open/close 3188 (time 1761316508.63 total 40.03 last 81.50) - open/close 3941 (time 1761316518.64 total 50.04 last 75.21) - open/close 4736 (time 1761316528.65 total 60.05 last 79.45) - open/close 5542 (time 1761316538.65 total 70.05 last 80.59) - open/close 6295 (time 1761316548.67 total 80.07 last 75.14) - open/close 7019 (time 1761316558.67 total 90.07 last 72.38) - open/close 7762 (time 1761316568.68 total 100.08 last 74.27) - open/close 8564 (time 1761316578.69 total 110.09 last 80.11) - open/close 9454 (time 1761316588.69 total 120.10 last 88.95) total: 10000 open/close in 126.32 seconds: 79.17 ops/second - unlinked 0 (time 1761316598 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 904 (time 1761316671.88 total 10.01 last 90.34) - open/close 1800 (time 1761316681.88 total 20.01 last 89.58) - open/close 2678 (time 1761316691.89 total 30.01 last 87.75) - open/close 3518 (time 1761316701.90 total 40.02 last 83.94) - open/close 4409 (time 1761316711.91 total 50.03 last 89.00) - open/close 5214 (time 1761316721.91 total 60.04 last 80.45) - open/close 6019 (time 1761316731.94 total 70.06 last 80.30) - open/close 6768 (time 1761316741.94 total 80.07 last 74.87) - open/close 7542 (time 1761316751.95 total 90.07 last 77.33) - open/close 8350 (time 1761316761.95 total 100.08 last 80.76) - open/close 9153 (time 1761316771.97 total 110.09 last 80.18) - open/close 9939 (time 1761316781.98 total 120.10 last 78.52) total: 10000 open/close in 121.05 seconds: 82.61 ops/second - unlinked 0 (time 1761316786 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 824 (time 1761316875.78 total 10.01 last 82.35) - open/close 1766 (time 1761316885.78 total 20.01 last 94.19) - open/close 2586 (time 1761316895.79 total 30.01 last 81.95) - open/close 3370 (time 1761316905.80 total 40.02 last 78.32) - open/close 4144 (time 1761316915.81 total 50.04 last 77.29) - open/close 4871 (time 1761316925.81 total 60.04 last 72.70) - open/close 5634 (time 1761316935.82 total 70.04 last 76.26) - open/close 6437 (time 1761316945.82 total 80.04 last 80.29) - open/close 7247 (time 1761316955.82 total 90.04 last 81.00) - open/close 8067 (time 1761316965.82 total 100.05 last 81.95) - open/close 8848 (time 1761316975.83 total 110.05 last 78.09) - open/close 9583 (time 1761316985.83 total 120.06 last 73.45) total: 10000 open/close in 125.64 seconds: 79.59 ops/second - unlinked 0 (time 1761316994 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 709 (time 1761317086.95 total 10.01 last 70.83) - open/close 1483 (time 1761317096.95 total 20.01 last 77.37) - open/close 2354 (time 1761317106.97 total 30.03 last 86.95) - open/close 3220 (time 1761317116.98 total 40.04 last 86.49) - open/close 4051 (time 1761317126.98 total 50.04 last 83.10) - open/close 4829 (time 1761317137.01 total 60.07 last 77.61) - open/close 5604 (time 1761317147.02 total 70.08 last 77.43) - open/close 6358 (time 1761317157.02 total 80.08 last 75.35) - open/close 7135 (time 1761317167.03 total 90.09 last 77.63) - open/close 7956 (time 1761317177.03 total 100.09 last 82.08) - open/close 8841 (time 1761317187.04 total 110.10 last 88.45) - open/close 9665 (time 1761317197.05 total 120.11 last 82.35) total: 10000 open/close in 124.68 seconds: 80.20 ops/second - unlinked 0 (time 1761317204 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 739 (time 1761317285.25 total 10.01 last 73.85) - open/close 1536 (time 1761317295.26 total 20.01 last 79.64) - open/close 2338 (time 1761317305.26 total 30.02 last 80.19) - open/close 3125 (time 1761317315.26 total 40.02 last 78.64) - open/close 3791 (time 1761317325.27 total 50.03 last 66.59) - open/close 4480 (time 1761317335.27 total 60.03 last 68.89) - open/close 5068 (time 1761317345.29 total 70.05 last 58.67) - open/close 5659 (time 1761317355.30 total 80.06 last 59.05) - open/close 6361 (time 1761317365.31 total 90.07 last 70.09) - open/close 7112 (time 1761317375.33 total 100.09 last 74.97) - open/close 7876 (time 1761317385.34 total 110.10 last 76.31) - open/close 8644 (time 1761317395.34 total 120.10 last 76.79) - open/close 9397 (time 1761317405.35 total 130.11 last 75.24) total: 10000 open/close in 138.43 seconds: 72.24 ops/second - unlinked 0 (time 1761317416 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 787 (time 1761317503.52 total 10.00 last 78.70) - open/close 1568 (time 1761317513.53 total 20.00 last 78.07) - open/close 2395 (time 1761317523.54 total 30.01 last 82.64) - open/close 3183 (time 1761317533.54 total 40.01 last 78.77) - open/close 3990 (time 1761317543.55 total 50.02 last 80.64) - open/close 4798 (time 1761317553.55 total 60.03 last 80.73) - open/close 5593 (time 1761317563.56 total 70.03 last 79.47) - open/close 6501 (time 1761317573.57 total 80.04 last 90.71) - open/close 7317 (time 1761317583.57 total 90.05 last 81.57) - open/close 8152 (time 1761317593.58 total 100.06 last 83.41) - open/close 8878 (time 1761317603.60 total 110.07 last 72.48) - open/close 9546 (time 1761317613.61 total 120.08 last 66.73) total: 10000 open/close in 126.45 seconds: 79.08 ops/second - unlinked 0 (time 1761317623 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 828 (time 1761317706.93 total 10.00 last 82.78) - open/close 1627 (time 1761317716.93 total 20.01 last 79.84) - open/close 2450 (time 1761317726.94 total 30.02 last 82.24) - open/close 3223 (time 1761317736.95 total 40.03 last 77.19) - open/close 4020 (time 1761317746.96 total 50.04 last 79.64) - open/close 4823 (time 1761317756.97 total 60.04 last 80.28) - open/close 5535 (time 1761317766.98 total 70.06 last 71.09) - open/close 6317 (time 1761317776.99 total 80.06 last 78.15) - open/close 7045 (time 1761317787.00 total 90.08 last 72.72) - open/close 7793 (time 1761317797.01 total 100.08 last 74.73) - open/close 8595 (time 1761317807.01 total 110.09 last 80.19) - open/close 9319 (time 1761317817.02 total 120.09 last 72.35) total: 10000 open/close in 129.20 seconds: 77.40 ops/second - unlinked 0 (time 1761317829 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 875 (time 1761317912.77 total 10.01 last 87.45) - open/close 1778 (time 1761317922.77 total 20.01 last 90.29) - open/close 2662 (time 1761317932.78 total 30.01 last 88.34) - open/close 3498 (time 1761317942.78 total 40.02 last 83.55) - open/close 4365 (time 1761317952.79 total 50.02 last 86.69) - open/close 5284 (time 1761317962.79 total 60.03 last 91.85) - open/close 6125 (time 1761317972.79 total 70.03 last 84.08) - open/close 6917 (time 1761317982.81 total 80.04 last 79.11) - open/close 7697 (time 1761317992.81 total 90.05 last 77.96) - open/close 8533 (time 1761318002.82 total 100.05 last 83.56) - open/close 9288 (time 1761318012.82 total 110.06 last 75.45) total: 10000 open/close in 119.82 seconds: 83.46 ops/second - unlinked 0 (time 1761318026 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 961 (time 1761318103.85 total 10.01 last 96.00) - open/close 1716 (time 1761318113.86 total 20.02 last 75.43) - open/close 2486 (time 1761318123.87 total 30.02 last 76.97) - open/close 3264 (time 1761318133.89 total 40.05 last 77.63) - open/close 4006 (time 1761318143.91 total 50.06 last 74.06) - open/close 4819 (time 1761318153.91 total 60.07 last 81.29) - open/close 5651 (time 1761318163.92 total 70.08 last 83.12) - open/close 6609 (time 1761318173.93 total 80.08 last 95.72) - open/close 7507 (time 1761318183.93 total 90.09 last 89.75) - open/close 8347 (time 1761318193.94 total 100.10 last 83.91) - open/close 9158 (time 1761318203.96 total 110.12 last 80.93) - open/close 9970 (time 1761318213.98 total 120.14 last 81.08) total: 10000 open/close in 120.50 seconds: 82.99 ops/second - unlinked 0 (time 1761318217 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg337-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg337-server: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg337-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre On OST0, 1333 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg337-server: LNET ready to unload oleg337-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2