== conf-sanity test 69: replace an OST with the same index ========================================================== 19:35:18 (1763080518) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 851 (time 1763080591.70 total 10.00 last 85.09) - open/close 1611 (time 1763080601.70 total 20.00 last 75.97) - open/close 2418 (time 1763080611.71 total 30.01 last 80.62) - open/close 3177 (time 1763080621.72 total 40.02 last 75.84) - open/close 3804 (time 1763080631.73 total 50.03 last 62.66) - open/close 4448 (time 1763080641.74 total 60.03 last 64.36) - open/close 5113 (time 1763080651.74 total 70.04 last 66.47) - open/close 5777 (time 1763080661.75 total 80.05 last 66.32) - open/close 6508 (time 1763080671.77 total 90.07 last 72.95) - open/close 7248 (time 1763080681.79 total 100.09 last 73.88) - open/close 8005 (time 1763080691.81 total 110.11 last 75.56) - open/close 8793 (time 1763080701.81 total 120.11 last 78.75) - open/close 9532 (time 1763080711.82 total 130.12 last 73.83) total: 10000 open/close in 136.67 seconds: 73.17 ops/second - unlinked 0 (time 1763080721 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 763 (time 1763080813.24 total 10.01 last 76.23) - open/close 1518 (time 1763080823.25 total 20.02 last 75.45) - open/close 2246 (time 1763080833.26 total 30.03 last 72.71) - open/close 3003 (time 1763080843.26 total 40.03 last 75.69) - open/close 3759 (time 1763080853.27 total 50.04 last 75.56) - open/close 4464 (time 1763080863.27 total 60.04 last 70.44) - open/close 5164 (time 1763080873.28 total 70.05 last 69.94) - open/close 5930 (time 1763080883.28 total 80.05 last 76.59) - open/close 6641 (time 1763080893.30 total 90.07 last 70.99) - open/close 7374 (time 1763080903.31 total 100.08 last 73.23) - open/close 8091 (time 1763080913.32 total 110.09 last 71.64) - open/close 8794 (time 1763080923.34 total 120.10 last 70.18) - open/close 9504 (time 1763080933.34 total 130.11 last 70.93) total: 10000 open/close in 136.80 seconds: 73.10 ops/second - unlinked 0 (time 1763080944 ; total 1 ; last 1) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 763 (time 1763081039.46 total 10.01 last 76.20) - open/close 1502 (time 1763081049.47 total 20.02 last 73.86) - open/close 2272 (time 1763081059.48 total 30.03 last 76.91) - open/close 3051 (time 1763081069.48 total 40.03 last 77.90) - open/close 3857 (time 1763081079.49 total 50.04 last 80.55) - open/close 4540 (time 1763081089.49 total 60.04 last 68.26) - open/close 5258 (time 1763081099.50 total 70.05 last 71.76) - open/close 5987 (time 1763081109.51 total 80.05 last 72.85) - open/close 6729 (time 1763081119.51 total 90.06 last 74.15) - open/close 7490 (time 1763081129.52 total 100.07 last 76.03) - open/close 8234 (time 1763081139.53 total 110.08 last 74.35) - open/close 8922 (time 1763081149.53 total 120.08 last 68.77) - open/close 9598 (time 1763081159.54 total 130.08 last 67.58) total: 10000 open/close in 135.75 seconds: 73.66 ops/second - unlinked 0 (time 1763081169 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 736 (time 1763081262.79 total 10.00 last 73.57) - open/close 1521 (time 1763081272.80 total 20.01 last 78.43) - open/close 2276 (time 1763081282.80 total 30.01 last 75.49) - open/close 3059 (time 1763081292.80 total 40.02 last 78.29) - open/close 3836 (time 1763081302.81 total 50.02 last 77.65) - open/close 4663 (time 1763081312.82 total 60.03 last 82.61) - open/close 5432 (time 1763081322.83 total 70.04 last 76.85) - open/close 6195 (time 1763081332.84 total 80.05 last 76.20) - open/close 6876 (time 1763081342.85 total 90.06 last 68.04) - open/close 7528 (time 1763081352.86 total 100.07 last 65.14) - open/close 8240 (time 1763081362.86 total 110.07 last 71.18) - open/close 9008 (time 1763081372.86 total 120.07 last 76.80) - open/close 9753 (time 1763081382.87 total 130.08 last 74.46) total: 10000 open/close in 133.58 seconds: 74.86 ops/second - unlinked 0 (time 1763081389 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 681 (time 1763081483.98 total 10.01 last 68.04) - open/close 1365 (time 1763081493.98 total 20.01 last 68.39) - open/close 2104 (time 1763081503.99 total 30.02 last 73.83) - open/close 2899 (time 1763081514.00 total 40.03 last 79.42) - open/close 3711 (time 1763081524.01 total 50.04 last 81.15) - open/close 4536 (time 1763081534.02 total 60.04 last 82.45) - open/close 5417 (time 1763081544.02 total 70.05 last 88.07) - open/close 6318 (time 1763081554.03 total 80.05 last 90.03) - open/close 7081 (time 1763081564.03 total 90.06 last 76.28) - open/close 7874 (time 1763081574.03 total 100.06 last 79.30) - open/close 8668 (time 1763081584.04 total 110.06 last 79.32) - open/close 9514 (time 1763081594.05 total 120.07 last 84.53) total: 10000 open/close in 125.22 seconds: 79.86 ops/second - unlinked 0 (time 1763081601 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 725 (time 1763081683.54 total 10.00 last 72.48) - open/close 1456 (time 1763081693.55 total 20.02 last 72.98) - open/close 2170 (time 1763081703.56 total 30.02 last 71.37) - open/close 2902 (time 1763081713.56 total 40.02 last 73.20) - open/close 3712 (time 1763081723.56 total 50.03 last 80.96) - open/close 4422 (time 1763081733.56 total 60.03 last 71.00) - open/close 5130 (time 1763081743.57 total 70.04 last 70.73) - open/close 5780 (time 1763081753.59 total 80.05 last 64.91) - open/close 6459 (time 1763081763.60 total 90.06 last 67.82) - open/close 7063 (time 1763081773.62 total 100.09 last 60.26) - open/close 7690 (time 1763081783.65 total 110.11 last 62.53) - open/close 8384 (time 1763081793.66 total 120.12 last 69.33) - open/close 9146 (time 1763081803.67 total 130.14 last 76.09) - open/close 9853 (time 1763081813.68 total 140.15 last 70.65) total: 10000 open/close in 142.43 seconds: 70.21 ops/second - unlinked 0 (time 1763081820 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 665 (time 1763081915.45 total 10.00 last 66.49) - open/close 1386 (time 1763081925.45 total 20.00 last 72.09) - open/close 2118 (time 1763081935.45 total 30.00 last 73.19) - open/close 2849 (time 1763081945.47 total 40.02 last 72.96) - open/close 3568 (time 1763081955.47 total 50.02 last 71.89) - open/close 4322 (time 1763081965.47 total 60.03 last 75.38) - open/close 5103 (time 1763081975.47 total 70.03 last 78.08) - open/close 5915 (time 1763081985.48 total 80.03 last 81.19) - open/close 6686 (time 1763081995.49 total 90.04 last 77.02) - open/close 7421 (time 1763082005.49 total 100.05 last 73.46) - open/close 8154 (time 1763082015.51 total 110.06 last 73.17) - open/close 8912 (time 1763082025.51 total 120.07 last 75.76) - open/close 9672 (time 1763082035.52 total 130.08 last 75.91) total: 10000 open/close in 134.29 seconds: 74.46 ops/second - unlinked 0 (time 1763082043 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 834 (time 1763082135.37 total 10.01 last 83.35) - open/close 1665 (time 1763082145.38 total 20.02 last 83.01) - open/close 2453 (time 1763082155.38 total 30.02 last 78.76) - open/close 3230 (time 1763082165.39 total 40.03 last 77.62) - open/close 4064 (time 1763082175.40 total 50.04 last 83.37) - open/close 4875 (time 1763082185.40 total 60.05 last 81.03) - open/close 5660 (time 1763082195.43 total 70.07 last 78.32) - open/close 6418 (time 1763082205.43 total 80.07 last 75.75) - open/close 7229 (time 1763082215.44 total 90.08 last 81.04) - open/close 8017 (time 1763082225.45 total 100.09 last 78.71) - open/close 8757 (time 1763082235.46 total 110.10 last 73.94) - open/close 9521 (time 1763082245.47 total 120.11 last 76.37) total: 10000 open/close in 126.16 seconds: 79.27 ops/second - unlinked 0 (time 1763082254 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 747 (time 1763082355.67 total 10.00 last 74.67) - open/close 1493 (time 1763082365.68 total 20.02 last 74.48) - open/close 2197 (time 1763082375.69 total 30.02 last 70.36) - open/close 2912 (time 1763082385.70 total 40.04 last 71.41) - open/close 3676 (time 1763082395.72 total 50.06 last 76.25) - open/close 4389 (time 1763082405.73 total 60.07 last 71.19) - open/close 5126 (time 1763082415.74 total 70.08 last 73.64) - open/close 5822 (time 1763082425.75 total 80.09 last 69.53) - open/close 6592 (time 1763082435.75 total 90.09 last 76.97) - open/close 7348 (time 1763082445.76 total 100.10 last 75.56) - open/close 8107 (time 1763082455.77 total 110.11 last 75.81) - open/close 8887 (time 1763082465.78 total 120.12 last 77.95) - open/close 9615 (time 1763082475.79 total 130.13 last 72.72) total: 10000 open/close in 135.96 seconds: 73.55 ops/second - unlinked 0 (time 1763082485 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 805 (time 1763082576.75 total 10.01 last 80.43) - open/close 1586 (time 1763082586.77 total 20.02 last 77.99) - open/close 2384 (time 1763082596.77 total 30.02 last 79.80) - open/close 3204 (time 1763082606.77 total 40.03 last 81.98) - open/close 4001 (time 1763082616.78 total 50.04 last 79.62) - open/close 4737 (time 1763082626.79 total 60.04 last 73.56) - open/close 5493 (time 1763082636.79 total 70.05 last 75.56) - open/close 6266 (time 1763082646.80 total 80.05 last 77.27) - open/close 7011 (time 1763082656.80 total 90.05 last 74.49) - open/close 7972 (time 1763082666.80 total 100.06 last 96.04) - open/close 8754 (time 1763082676.81 total 110.06 last 78.18) - open/close 9504 (time 1763082686.82 total 120.08 last 74.87) total: 10000 open/close in 126.53 seconds: 79.04 ops/second - unlinked 0 (time 1763082696 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg327-server: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg327-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.127@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.127@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.127@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg327-server: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg327-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg327-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2