== conf-sanity test 69: replace an OST with the same index ========================================================== 01:24:17 (1769754257) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 715 (time 1769754343.22 total 10.01 last 71.43) - open/close 1577 (time 1769754353.23 total 20.02 last 86.15) - open/close 2344 (time 1769754363.24 total 30.02 last 76.64) - open/close 3266 (time 1769754373.24 total 40.03 last 92.17) - open/close 4118 (time 1769754383.25 total 50.03 last 85.15) - open/close 4959 (time 1769754393.25 total 60.04 last 84.06) - open/close 5691 (time 1769754403.26 total 70.04 last 73.16) - open/close 6576 (time 1769754413.29 total 80.07 last 88.23) - open/close 7184 (time 1769754423.29 total 90.07 last 60.79) - open/close 7995 (time 1769754433.30 total 100.09 last 81.00) - open/close 8825 (time 1769754443.30 total 110.09 last 82.99) - open/close 9578 (time 1769754453.31 total 120.09 last 75.23) total: 10000 open/close in 125.81 seconds: 79.49 ops/second - unlinked 0 (time 1769754463 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 783 (time 1769754565.85 total 10.01 last 78.24) - open/close 1418 (time 1769754575.86 total 20.03 last 63.37) - open/close 2075 (time 1769754585.87 total 30.03 last 65.69) - open/close 2874 (time 1769754595.88 total 40.04 last 79.81) - open/close 3634 (time 1769754605.88 total 50.04 last 75.99) - open/close 4513 (time 1769754615.88 total 60.05 last 87.85) - open/close 5416 (time 1769754625.89 total 70.05 last 90.25) - open/close 6260 (time 1769754635.90 total 80.06 last 84.33) - open/close 7018 (time 1769754645.92 total 90.08 last 75.66) - open/close 7718 (time 1769754655.93 total 100.09 last 69.91) - open/close 8541 (time 1769754665.93 total 110.09 last 82.29) - open/close 9449 (time 1769754675.95 total 120.11 last 90.66) total: 10000 open/close in 126.22 seconds: 79.22 ops/second - unlinked 0 (time 1769754687 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 760 (time 1769754786.97 total 10.02 last 75.88) - open/close 1530 (time 1769754796.98 total 20.02 last 76.93) - open/close 2197 (time 1769754806.99 total 30.03 last 66.67) - open/close 3030 (time 1769754816.99 total 40.03 last 83.29) - open/close 3823 (time 1769754827.00 total 50.04 last 79.19) - open/close 4525 (time 1769754837.02 total 60.06 last 70.09) - open/close 5149 (time 1769754847.02 total 70.06 last 62.37) - open/close 5811 (time 1769754857.03 total 80.07 last 66.14) - open/close 6535 (time 1769754867.04 total 90.08 last 72.36) - open/close 7143 (time 1769754877.04 total 100.08 last 60.79) - open/close 7862 (time 1769754887.05 total 110.09 last 71.83) - open/close 8573 (time 1769754897.07 total 120.11 last 70.93) - open/close 9469 (time 1769754907.09 total 130.13 last 89.48) total: 10000 open/close in 136.21 seconds: 73.42 ops/second - unlinked 0 (time 1769754917 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 708 (time 1769755008.95 total 10.01 last 70.70) - open/close 1321 (time 1769755018.95 total 20.02 last 61.27) - open/close 2016 (time 1769755028.96 total 30.03 last 69.45) - open/close 2779 (time 1769755038.96 total 40.03 last 76.28) - open/close 3622 (time 1769755048.96 total 50.03 last 84.29) - open/close 4530 (time 1769755058.96 total 60.03 last 90.80) - open/close 5481 (time 1769755068.97 total 70.04 last 95.03) - open/close 6355 (time 1769755078.98 total 80.04 last 87.36) - open/close 7226 (time 1769755088.98 total 90.05 last 87.08) - open/close 8072 (time 1769755098.98 total 100.05 last 84.55) - open/close 8821 (time 1769755108.99 total 110.06 last 74.85) - open/close 9577 (time 1769755119.01 total 120.08 last 75.44) total: 10000 open/close in 125.17 seconds: 79.89 ops/second - unlinked 0 (time 1769755129 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 705 (time 1769755230.76 total 10.01 last 70.46) - open/close 1512 (time 1769755240.76 total 20.01 last 80.67) - open/close 2341 (time 1769755250.77 total 30.02 last 82.84) - open/close 3046 (time 1769755260.77 total 40.02 last 70.48) - open/close 3774 (time 1769755270.77 total 50.02 last 72.79) - open/close 4579 (time 1769755280.79 total 60.04 last 80.38) - open/close 5478 (time 1769755290.79 total 70.04 last 89.87) - open/close 6174 (time 1769755300.80 total 80.05 last 69.54) - open/close 7044 (time 1769755310.80 total 90.05 last 86.96) - open/close 7871 (time 1769755320.81 total 100.06 last 82.64) - open/close 8497 (time 1769755330.82 total 110.06 last 62.57) - open/close 9068 (time 1769755340.82 total 120.07 last 57.06) - open/close 9758 (time 1769755350.83 total 130.08 last 68.93) total: 10000 open/close in 133.56 seconds: 74.87 ops/second - unlinked 0 (time 1769755358 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 722 (time 1769755451.38 total 10.01 last 72.12) - open/close 1263 (time 1769755461.40 total 20.02 last 54.03) - open/close 1925 (time 1769755471.40 total 30.03 last 66.17) - open/close 2729 (time 1769755481.40 total 40.03 last 80.37) - open/close 3449 (time 1769755491.41 total 50.04 last 71.93) - open/close 4200 (time 1769755501.42 total 60.04 last 75.09) - open/close 4937 (time 1769755511.42 total 70.05 last 73.65) - open/close 5563 (time 1769755521.44 total 80.07 last 62.50) - open/close 6063 (time 1769755531.44 total 90.07 last 49.97) - open/close 6736 (time 1769755541.45 total 100.08 last 67.24) - open/close 7418 (time 1769755551.46 total 110.09 last 68.12) - open/close 8038 (time 1769755561.47 total 120.10 last 61.97) - open/close 8749 (time 1769755571.48 total 130.11 last 71.02) - open/close 9424 (time 1769755581.49 total 140.12 last 67.47) total: 10000 open/close in 147.21 seconds: 67.93 ops/second - unlinked 0 (time 1769755593 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 775 (time 1769755692.97 total 10.01 last 77.46) - open/close 1679 (time 1769755702.97 total 20.01 last 90.39) - open/close 2545 (time 1769755712.98 total 30.01 last 86.53) - open/close 3426 (time 1769755722.99 total 40.03 last 88.00) - open/close 4273 (time 1769755732.99 total 50.03 last 84.67) - open/close 5220 (time 1769755742.99 total 60.03 last 94.69) - open/close 6081 (time 1769755753.00 total 70.04 last 86.03) - open/close 6779 (time 1769755763.01 total 80.04 last 69.77) - open/close 7618 (time 1769755773.01 total 90.05 last 83.84) - open/close 8430 (time 1769755783.02 total 100.06 last 81.13) - open/close 9295 (time 1769755793.03 total 110.06 last 86.45) total: 10000 open/close in 118.27 seconds: 84.56 ops/second - unlinked 0 (time 1769755805 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 859 (time 1769755895.17 total 10.00 last 85.89) - open/close 1676 (time 1769755905.17 total 20.00 last 81.67) - open/close 2569 (time 1769755915.18 total 30.01 last 89.22) - open/close 3390 (time 1769755925.19 total 40.02 last 82.05) - open/close 4213 (time 1769755935.20 total 50.03 last 82.20) - open/close 4988 (time 1769755945.20 total 60.03 last 77.47) - open/close 5863 (time 1769755955.22 total 70.05 last 87.40) - open/close 6709 (time 1769755965.22 total 80.05 last 84.54) - open/close 7556 (time 1769755975.23 total 90.06 last 84.61) - open/close 8394 (time 1769755985.24 total 100.07 last 83.75) - open/close 9163 (time 1769755995.25 total 110.08 last 76.83) total: 10000 open/close in 119.36 seconds: 83.78 ops/second - unlinked 0 (time 1769756008 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 829 (time 1769756112.23 total 10.01 last 82.83) - open/close 1441 (time 1769756122.24 total 20.01 last 61.17) - open/close 2100 (time 1769756132.25 total 30.03 last 65.81) - open/close 2793 (time 1769756142.27 total 40.05 last 69.16) - open/close 3385 (time 1769756152.28 total 50.06 last 59.14) - open/close 4206 (time 1769756162.28 total 60.06 last 82.07) - open/close 4982 (time 1769756172.29 total 70.07 last 77.57) - open/close 5836 (time 1769756182.29 total 80.07 last 85.35) - open/close 6605 (time 1769756192.30 total 90.08 last 76.85) - open/close 7282 (time 1769756202.30 total 100.08 last 67.67) - open/close 7780 (time 1769756212.31 total 110.09 last 49.77) - open/close 8474 (time 1769756222.32 total 120.10 last 69.33) - open/close 9167 (time 1769756232.34 total 130.12 last 69.18) - open/close 9777 (time 1769756242.34 total 140.12 last 60.99) total: 10000 open/close in 143.50 seconds: 69.69 ops/second - unlinked 0 (time 1769756250 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 707 (time 1769756345.78 total 10.00 last 70.68) - open/close 1521 (time 1769756355.80 total 20.02 last 81.25) - open/close 2381 (time 1769756365.80 total 30.03 last 85.95) - open/close 3137 (time 1769756375.81 total 40.04 last 75.53) - open/close 4020 (time 1769756385.82 total 50.04 last 88.25) - open/close 4836 (time 1769756395.82 total 60.05 last 81.57) - open/close 5746 (time 1769756405.84 total 70.06 last 90.84) - open/close 6537 (time 1769756415.84 total 80.06 last 79.09) - open/close 7296 (time 1769756425.86 total 90.08 last 75.80) - open/close 8122 (time 1769756435.87 total 100.09 last 82.52) - open/close 8830 (time 1769756445.88 total 110.10 last 70.73) - open/close 9738 (time 1769756455.88 total 120.10 last 90.77) total: 10000 open/close in 123.04 seconds: 81.27 ops/second - unlinked 0 (time 1769756463 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.143@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.143@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.143@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.143@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.143@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg243-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2