== conf-sanity test 69: replace an OST with the same index ========================================================== 14:51:49 (1761245509) start mds service on oleg115-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg115-server: Force libcfs to create 2 CPU partitions oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 734 (time 1761245582.62 total 10.00 last 73.37) - open/close 1466 (time 1761245592.63 total 20.01 last 73.17) - open/close 2267 (time 1761245602.64 total 30.02 last 79.99) - open/close 3049 (time 1761245612.65 total 40.04 last 78.08) - open/close 3793 (time 1761245622.66 total 50.04 last 74.37) - open/close 4537 (time 1761245632.66 total 60.05 last 74.36) - open/close 5222 (time 1761245642.67 total 70.05 last 68.48) - open/close 5978 (time 1761245652.67 total 80.05 last 75.57) - open/close 6666 (time 1761245662.68 total 90.06 last 68.73) - open/close 7443 (time 1761245672.68 total 100.06 last 77.69) - open/close 8171 (time 1761245682.69 total 110.07 last 72.74) - open/close 8848 (time 1761245692.70 total 120.08 last 67.64) - open/close 9522 (time 1761245702.70 total 130.09 last 67.36) total: 10000 open/close in 136.67 seconds: 73.17 ops/second - unlinked 0 (time 1761245712 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 815 (time 1761245801.05 total 10.00 last 81.50) - open/close 1326 (time 1761245811.07 total 20.02 last 51.01) - open/close 2040 (time 1761245821.08 total 30.03 last 71.30) - open/close 2818 (time 1761245831.09 total 40.03 last 77.78) - open/close 3607 (time 1761245841.09 total 50.04 last 78.88) - open/close 4385 (time 1761245851.09 total 60.04 last 77.76) - open/close 5170 (time 1761245861.10 total 70.05 last 78.43) - open/close 5908 (time 1761245871.11 total 80.05 last 73.77) - open/close 6567 (time 1761245881.11 total 90.06 last 65.87) - open/close 7301 (time 1761245891.11 total 100.06 last 73.39) - open/close 8016 (time 1761245901.12 total 110.07 last 71.45) - open/close 8711 (time 1761245911.13 total 120.08 last 69.43) - open/close 9316 (time 1761245921.15 total 130.10 last 60.35) - open/close 9988 (time 1761245931.17 total 140.12 last 67.10) total: 10000 open/close in 140.30 seconds: 71.28 ops/second - unlinked 0 (time 1761245934 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 735 (time 1761246021.48 total 10.01 last 73.44) - open/close 1502 (time 1761246031.48 total 20.02 last 76.63) - open/close 2272 (time 1761246041.49 total 30.03 last 76.93) - open/close 3001 (time 1761246051.51 total 40.04 last 72.79) - open/close 3640 (time 1761246061.52 total 50.05 last 63.85) - open/close 4259 (time 1761246071.52 total 60.05 last 61.89) - open/close 4997 (time 1761246081.52 total 70.06 last 73.76) - open/close 5695 (time 1761246091.53 total 80.06 last 69.77) - open/close 6418 (time 1761246101.54 total 90.07 last 72.24) - open/close 7175 (time 1761246111.54 total 100.07 last 75.65) - open/close 7904 (time 1761246121.54 total 110.08 last 72.88) - open/close 8635 (time 1761246131.55 total 120.08 last 73.07) - open/close 9327 (time 1761246141.56 total 130.09 last 69.13) total: 10000 open/close in 139.87 seconds: 71.50 ops/second - unlinked 0 (time 1761246155 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 694 (time 1761246253.54 total 10.00 last 69.38) - open/close 1470 (time 1761246263.55 total 20.01 last 77.57) - open/close 2278 (time 1761246273.55 total 30.01 last 80.80) - open/close 3011 (time 1761246283.57 total 40.02 last 73.17) - open/close 3728 (time 1761246293.57 total 50.03 last 71.64) - open/close 4493 (time 1761246303.58 total 60.04 last 76.45) - open/close 5298 (time 1761246313.59 total 70.05 last 80.41) - open/close 6110 (time 1761246323.61 total 80.07 last 81.02) - open/close 6848 (time 1761246333.62 total 90.08 last 73.76) - open/close 7670 (time 1761246343.63 total 100.09 last 82.10) - open/close 8416 (time 1761246353.63 total 110.09 last 74.60) - open/close 9168 (time 1761246363.63 total 120.09 last 75.19) - open/close 9940 (time 1761246373.64 total 130.10 last 77.12) total: 10000 open/close in 130.84 seconds: 76.43 ops/second - unlinked 0 (time 1761246377 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 724 (time 1761246466.44 total 10.01 last 72.34) - open/close 1486 (time 1761246476.45 total 20.02 last 76.12) - open/close 2245 (time 1761246486.45 total 30.03 last 75.85) - open/close 3030 (time 1761246496.47 total 40.04 last 78.37) - open/close 3812 (time 1761246506.47 total 50.04 last 78.18) - open/close 4579 (time 1761246516.48 total 60.05 last 76.67) - open/close 5331 (time 1761246526.49 total 70.06 last 75.08) - open/close 6170 (time 1761246536.50 total 80.07 last 83.81) - open/close 6971 (time 1761246546.50 total 90.07 last 80.10) - open/close 7752 (time 1761246556.51 total 100.08 last 78.04) - open/close 8537 (time 1761246566.53 total 110.10 last 78.38) - open/close 9370 (time 1761246576.53 total 120.10 last 83.27) total: 10000 open/close in 128.28 seconds: 77.96 ops/second - unlinked 0 (time 1761246588 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 733 (time 1761246678.83 total 10.01 last 73.24) - open/close 1512 (time 1761246688.84 total 20.02 last 77.84) - open/close 2311 (time 1761246698.85 total 30.03 last 79.82) - open/close 3061 (time 1761246708.85 total 40.03 last 74.99) - open/close 3826 (time 1761246718.86 total 50.03 last 76.49) - open/close 4751 (time 1761246728.86 total 60.03 last 92.49) - open/close 5631 (time 1761246738.86 total 70.04 last 87.95) - open/close 6400 (time 1761246748.87 total 80.04 last 76.88) - open/close 7187 (time 1761246758.87 total 90.04 last 78.68) - open/close 7927 (time 1761246768.88 total 100.05 last 73.92) - open/close 8695 (time 1761246778.89 total 110.07 last 76.69) - open/close 9484 (time 1761246788.90 total 120.07 last 78.84) total: 10000 open/close in 127.53 seconds: 78.41 ops/second - unlinked 0 (time 1761246800 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 732 (time 1761246893.58 total 10.01 last 73.13) - open/close 1545 (time 1761246903.58 total 20.01 last 81.30) - open/close 2323 (time 1761246913.59 total 30.02 last 77.74) - open/close 3264 (time 1761246923.60 total 40.03 last 93.99) - open/close 4059 (time 1761246933.60 total 50.03 last 79.48) - open/close 4902 (time 1761246943.61 total 60.05 last 84.18) - open/close 5725 (time 1761246953.62 total 70.05 last 82.23) - open/close 6522 (time 1761246963.63 total 80.06 last 79.65) - open/close 7327 (time 1761246973.63 total 90.06 last 80.49) - open/close 8127 (time 1761246983.63 total 100.06 last 79.98) - open/close 8962 (time 1761246993.64 total 110.07 last 83.45) - open/close 9720 (time 1761247003.64 total 120.08 last 75.76) total: 10000 open/close in 123.69 seconds: 80.85 ops/second - unlinked 0 (time 1761247010 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 842 (time 1761247089.19 total 10.00 last 84.16) - open/close 1724 (time 1761247099.19 total 20.01 last 88.19) - open/close 2529 (time 1761247109.20 total 30.02 last 80.42) - open/close 3354 (time 1761247119.21 total 40.02 last 82.45) - open/close 4116 (time 1761247129.22 total 50.03 last 76.14) - open/close 4867 (time 1761247139.23 total 60.04 last 74.99) - open/close 5661 (time 1761247149.23 total 70.05 last 79.39) - open/close 6467 (time 1761247159.24 total 80.05 last 80.55) - open/close 7274 (time 1761247169.24 total 90.05 last 80.68) - open/close 8070 (time 1761247179.25 total 100.06 last 79.55) - open/close 8845 (time 1761247189.26 total 110.07 last 77.40) - open/close 9523 (time 1761247199.26 total 120.07 last 67.80) total: 10000 open/close in 126.71 seconds: 78.92 ops/second - unlinked 0 (time 1761247209 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 790 (time 1761247293.96 total 10.02 last 78.87) - open/close 1603 (time 1761247303.97 total 20.02 last 81.27) - open/close 2381 (time 1761247313.97 total 30.02 last 77.78) - open/close 3153 (time 1761247323.98 total 40.03 last 77.10) - open/close 4004 (time 1761247333.99 total 50.04 last 85.03) - open/close 4995 (time 1761247344.01 total 60.06 last 98.94) - open/close 5924 (time 1761247354.01 total 70.06 last 92.87) - open/close 6737 (time 1761247364.02 total 80.08 last 81.19) - open/close 7520 (time 1761247374.03 total 90.08 last 78.29) - open/close 8368 (time 1761247384.03 total 100.08 last 84.78) - open/close 9194 (time 1761247394.03 total 110.08 last 82.60) total: 10000 open/close in 119.95 seconds: 83.37 ops/second - unlinked 0 (time 1761247407 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 783 (time 1761247493.00 total 10.01 last 78.24) - open/close 1565 (time 1761247503.01 total 20.01 last 78.16) - open/close 2315 (time 1761247513.02 total 30.02 last 74.94) - open/close 3081 (time 1761247523.02 total 40.03 last 76.57) - open/close 3868 (time 1761247533.03 total 50.03 last 78.64) - open/close 4642 (time 1761247543.03 total 60.04 last 77.38) - open/close 5409 (time 1761247553.04 total 70.04 last 76.66) - open/close 6105 (time 1761247563.05 total 80.05 last 69.51) - open/close 6814 (time 1761247573.06 total 90.06 last 70.82) - open/close 7607 (time 1761247583.07 total 100.07 last 79.25) - open/close 8405 (time 1761247593.07 total 110.08 last 79.73) - open/close 9171 (time 1761247603.08 total 120.09 last 76.53) total: 10000 open/close in 129.79 seconds: 77.05 ops/second - unlinked 0 (time 1761247616 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.115@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.115@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg115-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre On OST0, 13269 used inodes rc=0 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.115@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.115@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.115@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.115@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.201.115@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg115-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg115-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2