== conf-sanity test 69: replace an OST with the same index ========================================================== 04:41:26 (1769074886) start mds service on oleg238-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 634 (time 1769074983.55 total 10.00 last 63.38) - open/close 1220 (time 1769074993.57 total 20.03 last 58.44) - open/close 1787 (time 1769075003.58 total 30.04 last 56.67) - open/close 2480 (time 1769075013.58 total 40.04 last 69.26) - open/close 3153 (time 1769075023.59 total 50.05 last 67.26) - open/close 3881 (time 1769075033.59 total 60.05 last 72.78) - open/close 4397 (time 1769075043.60 total 70.06 last 51.58) - open/close 4949 (time 1769075053.61 total 80.06 last 55.16) - open/close 5475 (time 1769075063.62 total 90.07 last 52.55) - open/close 6041 (time 1769075073.62 total 100.07 last 56.60) - open/close 6769 (time 1769075083.62 total 110.08 last 72.74) - open/close 7452 (time 1769075093.63 total 120.09 last 68.27) - open/close 8066 (time 1769075103.63 total 130.09 last 61.40) - open/close 8715 (time 1769075113.65 total 140.11 last 64.75) - open/close 9301 (time 1769075123.66 total 150.12 last 58.54) - open/close 9887 (time 1769075133.67 total 160.13 last 58.51) total: 10000 open/close in 162.65 seconds: 61.48 ops/second - unlinked 0 (time 1769075142 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 713 (time 1769075253.80 total 10.01 last 71.21) - open/close 1331 (time 1769075263.81 total 20.03 last 61.72) - open/close 1963 (time 1769075273.82 total 30.03 last 63.16) - open/close 2646 (time 1769075283.82 total 40.03 last 68.29) - open/close 3290 (time 1769075293.83 total 50.04 last 64.34) - open/close 3903 (time 1769075303.84 total 60.06 last 61.21) - open/close 4529 (time 1769075313.86 total 70.07 last 62.51) - open/close 5251 (time 1769075323.86 total 80.08 last 72.17) - open/close 5851 (time 1769075333.88 total 90.09 last 59.91) - open/close 6432 (time 1769075343.89 total 100.11 last 58.01) - open/close 6940 (time 1769075353.89 total 110.11 last 50.79) - open/close 7434 (time 1769075363.91 total 120.12 last 49.33) - open/close 7886 (time 1769075373.91 total 130.13 last 45.18) - open/close 8383 (time 1769075383.92 total 140.14 last 49.64) - open/close 8990 (time 1769075393.95 total 150.16 last 60.57) - open/close 9606 (time 1769075403.95 total 160.16 last 61.59) total: 10000 open/close in 166.96 seconds: 59.89 ops/second - unlinked 0 (time 1769075417 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 664 (time 1769075537.36 total 10.01 last 66.36) - open/close 1247 (time 1769075547.39 total 20.03 last 58.16) - open/close 1844 (time 1769075557.39 total 30.04 last 59.65) - open/close 2451 (time 1769075567.41 total 40.05 last 60.63) - open/close 3090 (time 1769075577.42 total 50.06 last 63.84) - open/close 3773 (time 1769075587.42 total 60.06 last 68.27) - open/close 4390 (time 1769075597.42 total 70.06 last 61.69) - open/close 5006 (time 1769075607.43 total 80.08 last 61.53) - open/close 5560 (time 1769075617.47 total 90.11 last 55.21) - open/close 6142 (time 1769075627.48 total 100.13 last 58.11) - open/close 6577 (time 1769075637.50 total 110.15 last 43.42) - open/close 7220 (time 1769075647.50 total 120.15 last 64.29) - open/close 7874 (time 1769075657.51 total 130.15 last 65.38) - open/close 8497 (time 1769075667.52 total 140.16 last 62.22) - open/close 8960 (time 1769075677.52 total 150.17 last 46.28) - open/close 9583 (time 1769075687.55 total 160.19 last 62.14) total: 10000 open/close in 165.90 seconds: 60.28 ops/second - unlinked 0 (time 1769075697 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 463 (time 1769075813.75 total 10.01 last 46.26) - open/close 1068 (time 1769075823.78 total 20.03 last 60.34) - open/close 1622 (time 1769075833.80 total 30.05 last 55.30) - open/close 2200 (time 1769075843.81 total 40.07 last 57.71) - open/close 2858 (time 1769075853.83 total 50.08 last 65.70) - open/close 3536 (time 1769075863.83 total 60.09 last 67.77) - open/close 4174 (time 1769075873.84 total 70.09 last 63.76) - open/close 4815 (time 1769075883.84 total 80.10 last 64.07) - open/close 5538 (time 1769075893.85 total 90.10 last 72.28) - open/close 6154 (time 1769075903.85 total 100.11 last 61.56) - open/close 6746 (time 1769075913.86 total 110.12 last 59.14) - open/close 7357 (time 1769075923.87 total 120.12 last 61.09) - open/close 7999 (time 1769075933.88 total 130.13 last 64.13) - open/close 8564 (time 1769075943.89 total 140.14 last 56.45) - open/close 9185 (time 1769075953.89 total 150.15 last 62.06) - open/close 9872 (time 1769075963.89 total 160.15 last 68.69) total: 10000 open/close in 161.90 seconds: 61.77 ops/second - unlinked 0 (time 1769075971 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 592 (time 1769076094.74 total 10.02 last 59.06) - open/close 1021 (time 1769076104.75 total 20.04 last 42.85) - open/close 1605 (time 1769076114.75 total 30.04 last 58.39) - open/close 1973 (time 1769076124.77 total 40.05 last 36.74) - open/close 2387 (time 1769076134.77 total 50.06 last 41.39) - open/close 2870 (time 1769076144.82 total 60.10 last 48.10) - open/close 3442 (time 1769076154.82 total 70.10 last 57.17) - open/close 3941 (time 1769076164.82 total 80.11 last 49.89) - open/close 4488 (time 1769076174.82 total 90.11 last 54.70) - open/close 5116 (time 1769076184.83 total 100.12 last 62.72) - open/close 5771 (time 1769076194.85 total 110.13 last 65.39) - open/close 6397 (time 1769076204.86 total 120.14 last 62.54) - open/close 7071 (time 1769076214.87 total 130.15 last 67.36) - open/close 7628 (time 1769076224.87 total 140.16 last 55.67) - open/close 8300 (time 1769076234.88 total 150.16 last 67.14) - open/close 8975 (time 1769076244.88 total 160.17 last 67.48) - open/close 9675 (time 1769076254.88 total 170.17 last 70.00) total: 10000 open/close in 175.42 seconds: 57.01 ops/second - unlinked 0 (time 1769076265 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 607 (time 1769076386.31 total 10.00 last 60.68) - open/close 1258 (time 1769076396.33 total 20.03 last 64.95) - open/close 1970 (time 1769076406.33 total 30.03 last 71.19) - open/close 2653 (time 1769076416.34 total 40.04 last 68.23) - open/close 3324 (time 1769076426.35 total 50.04 last 67.07) - open/close 4063 (time 1769076436.35 total 60.05 last 73.89) - open/close 4724 (time 1769076446.35 total 70.05 last 66.10) - open/close 5432 (time 1769076456.35 total 80.05 last 70.78) - open/close 6088 (time 1769076466.36 total 90.06 last 65.52) - open/close 6793 (time 1769076476.38 total 100.07 last 70.41) - open/close 7365 (time 1769076486.39 total 110.09 last 57.13) - open/close 8075 (time 1769076496.40 total 120.09 last 70.94) - open/close 8746 (time 1769076506.40 total 130.10 last 67.06) - open/close 9426 (time 1769076516.41 total 140.10 last 67.99) total: 10000 open/close in 148.39 seconds: 67.39 ops/second - unlinked 0 (time 1769076530 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 496 (time 1769076641.61 total 10.01 last 49.57) - open/close 1092 (time 1769076651.62 total 20.01 last 59.58) - open/close 1781 (time 1769076661.62 total 30.01 last 68.86) - open/close 2370 (time 1769076671.63 total 40.02 last 58.85) - open/close 2919 (time 1769076681.64 total 50.03 last 54.85) - open/close 3517 (time 1769076691.65 total 60.05 last 59.71) - open/close 4148 (time 1769076701.67 total 70.06 last 63.01) - open/close 4824 (time 1769076711.68 total 80.07 last 67.52) - open/close 5520 (time 1769076721.68 total 90.07 last 69.59) - open/close 6227 (time 1769076731.68 total 100.08 last 70.69) - open/close 6721 (time 1769076741.71 total 110.11 last 49.25) - open/close 7334 (time 1769076751.73 total 120.12 last 61.20) - open/close 7908 (time 1769076761.74 total 130.13 last 57.34) - open/close 8548 (time 1769076771.75 total 140.15 last 63.90) - open/close 9094 (time 1769076781.76 total 150.16 last 54.56) - open/close 9634 (time 1769076791.78 total 160.18 last 53.89) total: 10000 open/close in 166.31 seconds: 60.13 ops/second - unlinked 0 (time 1769076803 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 610 (time 1769076925.89 total 10.02 last 60.90) - open/close 1184 (time 1769076935.92 total 20.04 last 57.24) - open/close 1750 (time 1769076945.95 total 30.07 last 56.45) - open/close 2454 (time 1769076955.95 total 40.07 last 70.38) - open/close 3014 (time 1769076965.97 total 50.09 last 55.90) - open/close 3567 (time 1769076975.97 total 60.10 last 55.27) - open/close 4205 (time 1769076985.98 total 70.10 last 63.76) - open/close 4890 (time 1769076995.98 total 80.11 last 68.47) - open/close 5528 (time 1769077005.99 total 90.11 last 63.77) - open/close 6259 (time 1769077015.99 total 100.11 last 73.09) - open/close 6879 (time 1769077026.00 total 110.12 last 61.94) - open/close 7511 (time 1769077036.01 total 120.14 last 63.10) - open/close 8079 (time 1769077046.02 total 130.15 last 56.75) - open/close 8697 (time 1769077056.03 total 140.16 last 61.74) - open/close 9401 (time 1769077066.04 total 150.16 last 70.36) total: 10000 open/close in 158.46 seconds: 63.11 ops/second - unlinked 0 (time 1769077079 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 585 (time 1769077196.36 total 10.01 last 58.43) - open/close 1226 (time 1769077206.37 total 20.02 last 64.04) - open/close 1861 (time 1769077216.38 total 30.03 last 63.46) - open/close 2450 (time 1769077226.41 total 40.06 last 58.72) - open/close 3012 (time 1769077236.42 total 50.07 last 56.14) - open/close 3608 (time 1769077246.43 total 60.08 last 59.54) - open/close 4153 (time 1769077256.43 total 70.08 last 54.50) - open/close 4698 (time 1769077266.43 total 80.08 last 54.48) - open/close 5307 (time 1769077276.44 total 90.09 last 60.87) - open/close 5870 (time 1769077286.45 total 100.10 last 56.24) - open/close 6604 (time 1769077296.47 total 110.13 last 73.21) - open/close 7137 (time 1769077306.48 total 120.13 last 53.26) - open/close 7792 (time 1769077316.49 total 130.14 last 65.43) - open/close 8387 (time 1769077326.50 total 140.15 last 59.45) - open/close 9044 (time 1769077336.51 total 150.16 last 65.66) - open/close 9808 (time 1769077346.52 total 160.17 last 76.31) total: 10000 open/close in 164.24 seconds: 60.89 ops/second - unlinked 0 (time 1769077355 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 540 (time 1769077468.49 total 10.00 last 53.98) - open/close 1112 (time 1769077478.51 total 20.02 last 57.09) - open/close 1842 (time 1769077488.52 total 30.03 last 72.93) - open/close 2557 (time 1769077498.53 total 40.04 last 71.44) - open/close 3334 (time 1769077508.53 total 50.05 last 77.64) - open/close 4116 (time 1769077518.54 total 60.06 last 78.14) - open/close 4863 (time 1769077528.55 total 70.06 last 74.66) - open/close 5569 (time 1769077538.56 total 80.08 last 70.50) - open/close 6167 (time 1769077548.57 total 90.09 last 59.74) - open/close 6814 (time 1769077558.58 total 100.10 last 64.63) - open/close 7524 (time 1769077568.59 total 110.11 last 70.95) - open/close 8081 (time 1769077578.61 total 120.12 last 55.59) - open/close 8627 (time 1769077588.62 total 130.13 last 54.56) - open/close 9328 (time 1769077598.63 total 140.14 last 70.04) - open/close 9980 (time 1769077608.65 total 150.16 last 65.08) total: 10000 open/close in 150.42 seconds: 66.48 ops/second - unlinked 0 (time 1769077613 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.138@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.138@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.202.138@tcp autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=192.168.202.138@tcp autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:mgsnode=192.168.202.138@tcp lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2