== conf-sanity test 69: replace an OST with the same index ========================================================== 14:10:46 (1758737446) start mds service on oleg314-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg314-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg314-server: detected 4 online CPUs by sysfs oleg314-server: MODOPTS_LIBCFS= oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg314-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 800 (time 1758737513.87 total 10.01 last 79.91) - open/close 1539 (time 1758737523.87 total 20.02 last 73.86) - open/close 2342 (time 1758737533.88 total 30.02 last 80.24) - open/close 3146 (time 1758737543.89 total 40.03 last 80.32) - open/close 3855 (time 1758737553.91 total 50.06 last 70.74) - open/close 4533 (time 1758737563.93 total 60.07 last 67.69) - open/close 5276 (time 1758737573.94 total 70.08 last 74.24) - open/close 6089 (time 1758737583.95 total 80.09 last 81.20) - open/close 6947 (time 1758737593.96 total 90.10 last 85.71) - open/close 7839 (time 1758737603.96 total 100.10 last 89.18) - open/close 8703 (time 1758737613.97 total 110.11 last 86.36) - open/close 9540 (time 1758737623.97 total 120.12 last 83.64) total: 10000 open/close in 125.11 seconds: 79.93 ops/second - unlinked 0 (time 1758737631 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 730 (time 1758737708.96 total 10.00 last 72.98) - open/close 1628 (time 1758737718.97 total 20.01 last 89.74) - open/close 2577 (time 1758737728.98 total 30.02 last 94.81) - open/close 3423 (time 1758737738.99 total 40.03 last 84.51) - open/close 4314 (time 1758737748.99 total 50.03 last 89.06) - open/close 5455 (time 1758737759.00 total 60.04 last 114.02) - open/close 6716 (time 1758737769.00 total 70.04 last 126.06) - open/close 7641 (time 1758737779.01 total 80.05 last 92.42) - open/close 8823 (time 1758737789.01 total 90.06 last 118.14) - open/close 9878 (time 1758737799.02 total 100.06 last 105.48) total: 10000 open/close in 101.46 seconds: 98.56 ops/second - unlinked 0 (time 1758737802 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 850 (time 1758737866.48 total 10.00 last 84.99) - open/close 1723 (time 1758737876.48 total 20.01 last 87.26) - open/close 2491 (time 1758737886.49 total 30.02 last 76.72) - open/close 3289 (time 1758737896.50 total 40.02 last 79.75) - open/close 4143 (time 1758737906.50 total 50.02 last 85.38) - open/close 5227 (time 1758737916.51 total 60.03 last 108.34) - open/close 6238 (time 1758737926.51 total 70.03 last 101.06) - open/close 7081 (time 1758737936.52 total 80.05 last 84.20) - open/close 8285 (time 1758737946.52 total 90.05 last 120.38) - open/close 9419 (time 1758737956.52 total 100.05 last 113.39) total: 10000 open/close in 106.04 seconds: 94.30 ops/second - unlinked 0 (time 1758737964 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 833 (time 1758738039.82 total 10.01 last 83.21) - open/close 1741 (time 1758738049.82 total 20.01 last 90.77) - open/close 2646 (time 1758738059.83 total 30.02 last 90.48) - open/close 3602 (time 1758738069.83 total 40.02 last 95.51) - open/close 4607 (time 1758738079.84 total 50.03 last 100.46) - open/close 5498 (time 1758738089.84 total 60.03 last 89.05) - open/close 6544 (time 1758738099.85 total 70.04 last 104.58) - open/close 7976 (time 1758738109.86 total 80.05 last 143.05) - open/close 9308 (time 1758738119.86 total 90.05 last 133.13) total: 10000 open/close in 95.96 seconds: 104.21 ops/second - unlinked 0 (time 1758738128 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1045 (time 1758738204.05 total 10.00 last 104.48) - open/close 2060 (time 1758738214.05 total 20.00 last 101.47) - open/close 3502 (time 1758738224.06 total 30.01 last 144.15) - open/close 4953 (time 1758738234.06 total 40.01 last 145.06) - open/close 5989 (time 1758738244.07 total 50.02 last 103.52) - open/close 6962 (time 1758738254.07 total 60.03 last 97.23) - open/close 7824 (time 1758738264.07 total 70.03 last 86.18) - open/close 8807 (time 1758738274.08 total 80.03 last 98.27) - open/close 9753 (time 1758738284.08 total 90.03 last 94.59) total: 10000 open/close in 91.80 seconds: 108.94 ops/second - unlinked 0 (time 1758738287 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 915 (time 1758738354.80 total 10.01 last 91.39) - open/close 1767 (time 1758738364.81 total 20.02 last 85.16) - open/close 2705 (time 1758738374.81 total 30.02 last 93.78) - open/close 4029 (time 1758738384.81 total 40.02 last 132.37) - open/close 5181 (time 1758738394.82 total 50.03 last 115.11) - open/close 6222 (time 1758738404.83 total 60.04 last 104.02) - open/close 7384 (time 1758738414.83 total 70.04 last 116.18) - open/close 8556 (time 1758738424.84 total 80.05 last 117.11) - open/close 9502 (time 1758738434.85 total 90.05 last 94.53) total: 10000 open/close in 95.09 seconds: 105.16 ops/second - unlinked 0 (time 1758738442 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 838 (time 1758738520.75 total 10.01 last 83.71) - open/close 1703 (time 1758738530.76 total 20.02 last 86.46) - open/close 2530 (time 1758738540.76 total 30.02 last 82.65) - open/close 3317 (time 1758738550.77 total 40.03 last 78.62) - open/close 4279 (time 1758738560.78 total 50.04 last 96.13) - open/close 5289 (time 1758738570.78 total 60.04 last 100.96) - open/close 6153 (time 1758738580.79 total 70.04 last 86.38) - open/close 7035 (time 1758738590.79 total 80.05 last 88.16) - open/close 7883 (time 1758738600.81 total 90.07 last 84.63) - open/close 8682 (time 1758738610.81 total 100.07 last 79.89) - open/close 9448 (time 1758738620.81 total 110.07 last 76.57) total: 10000 open/close in 117.48 seconds: 85.12 ops/second - unlinked 0 (time 1758738631 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 911 (time 1758738708.99 total 10.01 last 91.05) - open/close 1771 (time 1758738719.00 total 20.02 last 85.89) - open/close 2609 (time 1758738729.01 total 30.03 last 83.69) - open/close 3426 (time 1758738739.01 total 40.03 last 81.70) - open/close 4255 (time 1758738749.01 total 50.03 last 82.89) - open/close 5046 (time 1758738759.02 total 60.04 last 79.07) - open/close 5863 (time 1758738769.02 total 70.04 last 81.66) - open/close 6668 (time 1758738779.03 total 80.05 last 80.45) - open/close 7420 (time 1758738789.03 total 90.05 last 75.15) - open/close 8152 (time 1758738799.03 total 100.05 last 73.20) - open/close 8952 (time 1758738809.04 total 110.06 last 79.98) - open/close 9722 (time 1758738819.05 total 120.07 last 76.92) total: 10000 open/close in 123.80 seconds: 80.77 ops/second - unlinked 0 (time 1758738826 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 847 (time 1758738916.83 total 10.00 last 84.68) - open/close 1724 (time 1758738926.84 total 20.01 last 87.61) - open/close 2553 (time 1758738936.84 total 30.01 last 82.90) - open/close 3389 (time 1758738946.85 total 40.02 last 83.55) - open/close 4114 (time 1758738956.86 total 50.03 last 72.45) - open/close 4928 (time 1758738966.87 total 60.03 last 81.32) - open/close 5720 (time 1758738976.87 total 70.04 last 79.16) - open/close 6516 (time 1758738986.88 total 80.05 last 79.53) - open/close 7347 (time 1758738996.89 total 90.06 last 83.04) - open/close 8127 (time 1758739006.89 total 100.06 last 77.95) - open/close 8906 (time 1758739016.90 total 110.07 last 77.84) - open/close 9703 (time 1758739026.91 total 120.08 last 79.61) total: 10000 open/close in 123.95 seconds: 80.68 ops/second - unlinked 0 (time 1758739034 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 830 (time 1758739124.97 total 10.01 last 82.91) - open/close 1715 (time 1758739134.97 total 20.01 last 88.49) - open/close 2565 (time 1758739144.97 total 30.01 last 84.98) - open/close 3349 (time 1758739154.98 total 40.02 last 78.33) - open/close 4144 (time 1758739164.98 total 50.03 last 79.47) - open/close 4942 (time 1758739175.01 total 60.05 last 79.58) - open/close 5750 (time 1758739185.02 total 70.06 last 80.74) - open/close 6551 (time 1758739195.02 total 80.06 last 80.09) - open/close 7269 (time 1758739205.02 total 90.07 last 71.78) - open/close 8015 (time 1758739215.04 total 100.08 last 74.51) - open/close 8712 (time 1758739225.04 total 110.09 last 69.64) - open/close 9505 (time 1758739235.05 total 120.09 last 79.27) total: 10000 open/close in 126.78 seconds: 78.88 ops/second - unlinked 0 (time 1758739244 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre On OST0, 8597 used inodes rc=0 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.114@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.114@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.114@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg314-server: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg314-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local oleg314-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg314-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2