== conf-sanity test 69: replace an OST with the same index ========================================================== 13:47:29 (1752688049) start mds service on oleg351-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1229 (time 1752688089.55 total 10.00 last 122.88) - open/close 2263 (time 1752688099.55 total 20.01 last 103.34) - open/close 3176 (time 1752688109.56 total 30.02 last 91.21) - open/close 4315 (time 1752688119.56 total 40.02 last 113.88) - open/close 5261 (time 1752688129.57 total 50.02 last 94.57) - open/close 6511 (time 1752688139.57 total 60.03 last 124.94) - open/close 8113 (time 1752688149.58 total 70.03 last 160.16) - open/close 9747 (time 1752688159.58 total 80.03 last 163.37) total: 10000 open/close in 81.84 seconds: 122.19 ops/second - unlinked 0 (time 1752688163 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1980 (time 1752688211.85 total 10.01 last 197.83) - open/close 4499 (time 1752688221.85 total 20.01 last 251.87) - open/close 6504 (time 1752688231.86 total 30.01 last 200.42) - open/close 8299 (time 1752688241.86 total 40.02 last 179.36) total: 10000 open/close in 48.34 seconds: 206.88 ops/second - unlinked 0 (time 1752688251 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1738 (time 1752688305.50 total 10.00 last 173.77) - open/close 3342 (time 1752688315.50 total 20.00 last 160.35) - open/close 5050 (time 1752688325.51 total 30.01 last 170.77) - open/close 6996 (time 1752688335.51 total 40.01 last 194.52) - open/close 8729 (time 1752688345.51 total 50.01 last 173.26) total: 10000 open/close in 57.95 seconds: 172.57 ops/second - unlinked 0 (time 1752688355 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1350 (time 1752688399.33 total 10.00 last 134.99) - open/close 2543 (time 1752688409.34 total 20.01 last 119.22) - open/close 3747 (time 1752688419.34 total 30.01 last 120.38) - open/close 5269 (time 1752688429.34 total 40.01 last 152.13) - open/close 7037 (time 1752688439.34 total 50.01 last 176.79) - open/close 8474 (time 1752688449.35 total 60.02 last 143.64) total: 10000 open/close in 69.21 seconds: 144.48 ops/second - unlinked 0 (time 1752688460 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1502 (time 1752688505.80 total 10.00 last 150.15) - open/close 3087 (time 1752688515.80 total 20.00 last 158.49) - open/close 4620 (time 1752688525.80 total 30.00 last 153.29) - open/close 6685 (time 1752688535.81 total 40.01 last 206.48) - open/close 8638 (time 1752688545.81 total 50.01 last 195.26) total: 10000 open/close in 55.95 seconds: 178.75 ops/second - unlinked 0 (time 1752688553 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2921 (time 1752688585.55 total 10.00 last 292.05) - open/close 5425 (time 1752688595.55 total 20.01 last 250.24) - open/close 8064 (time 1752688605.55 total 30.01 last 263.90) total: 10000 open/close in 35.91 seconds: 278.51 ops/second - unlinked 0 (time 1752688613 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2879 (time 1752688646.33 total 10.00 last 287.83) - open/close 5760 (time 1752688656.33 total 20.00 last 288.09) - open/close 8562 (time 1752688666.34 total 30.01 last 280.11) total: 10000 open/close in 34.71 seconds: 288.10 ops/second - unlinked 0 (time 1752688672 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2488 (time 1752688704.93 total 10.00 last 248.74) - open/close 5257 (time 1752688714.93 total 20.00 last 276.90) - open/close 8286 (time 1752688724.93 total 30.00 last 302.82) total: 10000 open/close in 36.47 seconds: 274.22 ops/second - unlinked 0 (time 1752688732 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 3019 (time 1752688766.36 total 10.00 last 301.79) - open/close 5846 (time 1752688776.37 total 20.01 last 282.61) - open/close 8518 (time 1752688786.37 total 30.01 last 267.15) total: 10000 open/close in 35.72 seconds: 279.99 ops/second - unlinked 0 (time 1752688793 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2839 (time 1752688825.03 total 10.00 last 283.82) - open/close 5745 (time 1752688835.03 total 20.01 last 290.52) - open/close 8833 (time 1752688845.03 total 30.01 last 308.78) total: 10000 open/close in 33.95 seconds: 294.58 ops/second - unlinked 0 (time 1752688850 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre On OST0, 18805 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2