== conf-sanity test 69: replace an OST with the same index ========================================================== 21:37:08 (1767839828) start mds service on oleg409-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2054 (time 1767839877.89 total 10.01 last 205.14) - open/close 3399 (time 1767839887.89 total 20.02 last 134.45) - open/close 4702 (time 1767839897.90 total 30.02 last 130.27) - open/close 6454 (time 1767839907.90 total 40.02 last 175.18) - open/close 7957 (time 1767839917.91 total 50.03 last 150.21) - open/close 9393 (time 1767839927.91 total 60.03 last 143.48) total: 10000 open/close in 64.74 seconds: 154.46 ops/second - unlinked 0 (time 1767839935 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1481 (time 1767839986.68 total 10.00 last 148.06) - open/close 2784 (time 1767839996.68 total 20.01 last 130.23) - open/close 4401 (time 1767840006.69 total 30.01 last 161.64) - open/close 6103 (time 1767840016.69 total 40.01 last 170.16) - open/close 7734 (time 1767840026.69 total 50.02 last 163.04) - open/close 9426 (time 1767840036.70 total 60.02 last 169.09) total: 10000 open/close in 63.54 seconds: 157.39 ops/second - unlinked 0 (time 1767840041 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1457 (time 1767840091.19 total 10.00 last 145.66) - open/close 3248 (time 1767840101.19 total 20.00 last 179.10) - open/close 4536 (time 1767840111.19 total 30.00 last 128.78) - open/close 5937 (time 1767840121.20 total 40.01 last 140.00) - open/close 7313 (time 1767840131.21 total 50.02 last 137.47) - open/close 8760 (time 1767840141.22 total 60.03 last 144.59) total: 10000 open/close in 68.96 seconds: 145.01 ops/second - unlinked 0 (time 1767840152 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1426 (time 1767840200.61 total 10.00 last 142.55) - open/close 2837 (time 1767840210.61 total 20.00 last 141.10) - open/close 4314 (time 1767840220.62 total 30.01 last 147.62) - open/close 5738 (time 1767840230.62 total 40.01 last 142.36) - open/close 7397 (time 1767840240.62 total 50.01 last 165.86) - open/close 8861 (time 1767840250.62 total 60.01 last 146.39) total: 10000 open/close in 68.40 seconds: 146.21 ops/second - unlinked 0 (time 1767840261 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1869 (time 1767840313.20 total 10.00 last 186.89) - open/close 4329 (time 1767840323.21 total 20.00 last 245.95) - open/close 6442 (time 1767840333.21 total 30.01 last 211.18) - open/close 8254 (time 1767840343.21 total 40.01 last 181.16) - open/close 9802 (time 1767840353.22 total 50.01 last 154.77) total: 10000 open/close in 51.17 seconds: 195.44 ops/second - unlinked 0 (time 1767840356 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 4326 (time 1767840397.86 total 10.00 last 432.56) - open/close 8676 (time 1767840407.86 total 20.00 last 434.88) total: 10000 open/close in 23.13 seconds: 432.36 ops/second - unlinked 0 (time 1767840412 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5028 (time 1767840436.26 total 10.00 last 502.75) total: 10000 open/close in 20.00 seconds: 500.10 ops/second - unlinked 0 (time 1767840447 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5055 (time 1767840470.72 total 10.00 last 505.48) total: 10000 open/close in 19.21 seconds: 520.55 ops/second - unlinked 0 (time 1767840481 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4491 (time 1767840504.12 total 10.00 last 449.06) - open/close 9748 (time 1767840514.12 total 20.00 last 525.69) total: 10000 open/close in 20.52 seconds: 487.38 ops/second - unlinked 0 (time 1767840515 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4987 (time 1767840538.69 total 10.00 last 498.67) total: 10000 open/close in 19.03 seconds: 525.56 ops/second - unlinked 0 (time 1767840548 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre On OST0, 4509 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg409-server: LNET ready to unload oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2