== conf-sanity test 69: replace an OST with the same index ========================================================== 09:46:48 (1753364808) start mds service on oleg201-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2021 (time 1753364852.11 total 10.00 last 202.10) - open/close 4083 (time 1753364862.11 total 20.01 last 206.10) - open/close 6521 (time 1753364872.12 total 30.01 last 243.69) - open/close 8679 (time 1753364882.12 total 40.01 last 215.69) total: 10000 open/close in 49.17 seconds: 203.38 ops/second - unlinked 0 (time 1753364892 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2196 (time 1753364929.86 total 10.00 last 219.58) - open/close 4552 (time 1753364939.86 total 20.00 last 235.59) - open/close 5946 (time 1753364949.87 total 30.01 last 139.28) - open/close 6764 (time 1753364959.88 total 40.02 last 81.73) - open/close 7580 (time 1753364969.88 total 50.02 last 81.59) - open/close 8440 (time 1753364979.88 total 60.02 last 85.99) - open/close 9563 (time 1753364989.89 total 70.03 last 112.18) total: 10000 open/close in 73.91 seconds: 135.30 ops/second - unlinked 0 (time 1753364995 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1352 (time 1753365043.47 total 10.01 last 135.02) - open/close 2691 (time 1753365053.47 total 20.02 last 133.83) - open/close 4241 (time 1753365063.47 total 30.02 last 155.00) - open/close 5575 (time 1753365073.47 total 40.02 last 133.38) - open/close 7116 (time 1753365083.48 total 50.02 last 154.07) - open/close 8470 (time 1753365093.48 total 60.03 last 135.34) - open/close 9748 (time 1753365103.48 total 70.03 last 127.77) total: 10000 open/close in 72.01 seconds: 138.87 ops/second - unlinked 0 (time 1753365107 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2198 (time 1753365150.84 total 10.00 last 219.74) - open/close 3906 (time 1753365160.84 total 20.01 last 170.75) - open/close 5444 (time 1753365170.84 total 30.01 last 153.79) - open/close 7177 (time 1753365180.84 total 40.01 last 173.29) - open/close 8969 (time 1753365190.85 total 50.02 last 179.01) total: 10000 open/close in 55.34 seconds: 180.70 ops/second - unlinked 0 (time 1753365197 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1927 (time 1753365230.96 total 10.00 last 192.65) - open/close 4022 (time 1753365240.97 total 20.01 last 209.40) - open/close 5997 (time 1753365250.97 total 30.01 last 197.44) - open/close 8108 (time 1753365260.97 total 40.01 last 211.10) total: 10000 open/close in 49.53 seconds: 201.89 ops/second - unlinked 0 (time 1753365272 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2070 (time 1753365307.56 total 10.00 last 206.99) - open/close 4482 (time 1753365317.56 total 20.01 last 241.03) - open/close 6760 (time 1753365327.57 total 30.01 last 227.77) - open/close 9057 (time 1753365337.57 total 40.01 last 229.69) total: 10000 open/close in 43.95 seconds: 227.54 ops/second - unlinked 0 (time 1753365343 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2696 (time 1753365372.10 total 10.00 last 269.57) - open/close 5309 (time 1753365382.11 total 20.00 last 261.22) - open/close 8274 (time 1753365392.11 total 30.01 last 296.41) total: 10000 open/close in 36.48 seconds: 274.13 ops/second - unlinked 0 (time 1753365400 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3556 (time 1753365432.59 total 10.00 last 355.50) - open/close 6940 (time 1753365442.59 total 20.01 last 338.31) - open/close 9634 (time 1753365452.60 total 30.01 last 269.32) total: 10000 open/close in 31.48 seconds: 317.66 ops/second - unlinked 0 (time 1753365455 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2749 (time 1753365488.10 total 10.00 last 274.87) - open/close 4989 (time 1753365498.10 total 20.00 last 223.93) - open/close 7530 (time 1753365508.10 total 30.01 last 254.02) total: 10000 open/close in 38.11 seconds: 262.37 ops/second - unlinked 0 (time 1753365517 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2697 (time 1753365547.19 total 10.00 last 269.65) - open/close 5583 (time 1753365557.19 total 20.00 last 288.60) - open/close 8865 (time 1753365567.19 total 30.00 last 328.16) total: 10000 open/close in 33.26 seconds: 300.63 ops/second - unlinked 0 (time 1753365571 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.101@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre On OST0, 18757 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2