== conf-sanity test 69: replace an OST with the same index ========================================================== 23:13:34 (1769055214) start mds service on oleg227-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 524 (time 1769055311.20 total 10.00 last 52.39) - open/close 1029 (time 1769055321.21 total 20.01 last 50.45) - open/close 1557 (time 1769055331.21 total 30.01 last 52.79) - open/close 2161 (time 1769055341.23 total 40.03 last 60.29) - open/close 2830 (time 1769055351.24 total 50.04 last 66.85) - open/close 3530 (time 1769055361.25 total 60.05 last 69.90) - open/close 4103 (time 1769055371.26 total 70.06 last 57.29) - open/close 4715 (time 1769055381.26 total 80.06 last 61.16) - open/close 5327 (time 1769055391.27 total 90.07 last 61.12) - open/close 5928 (time 1769055401.28 total 100.08 last 60.08) - open/close 6423 (time 1769055411.28 total 110.08 last 49.48) - open/close 6867 (time 1769055421.30 total 120.10 last 44.33) - open/close 7383 (time 1769055431.31 total 130.11 last 51.56) - open/close 7865 (time 1769055441.31 total 140.11 last 48.19) - open/close 8397 (time 1769055451.32 total 150.12 last 53.16) - open/close 8952 (time 1769055461.32 total 160.12 last 55.49) - open/close 9520 (time 1769055471.34 total 170.14 last 56.70) total: 10000 open/close in 178.10 seconds: 56.15 ops/second - unlinked 0 (time 1769055484 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 641 (time 1769055597.18 total 10.01 last 64.05) - open/close 1307 (time 1769055607.18 total 20.01 last 66.59) - open/close 1956 (time 1769055617.19 total 30.02 last 64.83) - open/close 2562 (time 1769055627.19 total 40.02 last 60.59) - open/close 3188 (time 1769055637.19 total 50.02 last 62.60) - open/close 3752 (time 1769055647.20 total 60.02 last 56.39) - open/close 4350 (time 1769055657.20 total 70.03 last 59.78) - open/close 5015 (time 1769055667.20 total 80.03 last 66.48) - open/close 5573 (time 1769055677.22 total 90.04 last 55.72) - open/close 6185 (time 1769055687.23 total 100.06 last 61.11) - open/close 6837 (time 1769055697.23 total 110.06 last 65.18) - open/close 7566 (time 1769055707.24 total 120.07 last 72.86) - open/close 8148 (time 1769055717.25 total 130.08 last 58.13) - open/close 8718 (time 1769055727.26 total 140.09 last 56.93) - open/close 9318 (time 1769055737.26 total 150.09 last 59.99) - open/close 9868 (time 1769055747.27 total 160.10 last 54.95) total: 10000 open/close in 162.38 seconds: 61.58 ops/second - unlinked 0 (time 1769055754 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 632 (time 1769055866.11 total 10.01 last 63.13) - open/close 1240 (time 1769055876.12 total 20.02 last 60.77) - open/close 1886 (time 1769055886.12 total 30.02 last 64.56) - open/close 2501 (time 1769055896.13 total 40.03 last 61.45) - open/close 3207 (time 1769055906.14 total 50.04 last 70.54) - open/close 3857 (time 1769055916.14 total 60.04 last 64.97) - open/close 4506 (time 1769055926.15 total 70.05 last 64.86) - open/close 5168 (time 1769055936.17 total 80.07 last 66.08) - open/close 5712 (time 1769055946.17 total 90.07 last 54.37) - open/close 6360 (time 1769055956.18 total 100.08 last 64.76) - open/close 6984 (time 1769055966.18 total 110.08 last 62.37) - open/close 7638 (time 1769055976.18 total 120.09 last 65.39) - open/close 8173 (time 1769055986.19 total 130.09 last 53.45) - open/close 8734 (time 1769055996.21 total 140.11 last 56.00) - open/close 9352 (time 1769056006.22 total 150.13 last 61.71) - open/close 9950 (time 1769056016.24 total 160.14 last 59.72) total: 10000 open/close in 161.23 seconds: 62.02 ops/second - unlinked 0 (time 1769056022 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 608 (time 1769056137.19 total 10.00 last 60.78) - open/close 1209 (time 1769056147.20 total 20.01 last 60.05) - open/close 1885 (time 1769056157.21 total 30.02 last 67.54) - open/close 2500 (time 1769056167.21 total 40.03 last 61.45) - open/close 3122 (time 1769056177.24 total 50.05 last 62.05) - open/close 3760 (time 1769056187.25 total 60.06 last 63.73) - open/close 4380 (time 1769056197.25 total 70.07 last 61.98) - open/close 4984 (time 1769056207.27 total 80.09 last 60.29) - open/close 5532 (time 1769056217.28 total 90.09 last 54.77) - open/close 5969 (time 1769056227.30 total 100.12 last 43.59) - open/close 6605 (time 1769056237.32 total 110.14 last 63.46) - open/close 7228 (time 1769056247.33 total 120.14 last 62.28) - open/close 7823 (time 1769056257.33 total 130.15 last 59.49) - open/close 8386 (time 1769056267.34 total 140.15 last 56.26) - open/close 8966 (time 1769056277.35 total 150.17 last 57.90) - open/close 9552 (time 1769056287.36 total 160.18 last 58.56) total: 10000 open/close in 168.40 seconds: 59.38 ops/second - unlinked 0 (time 1769056300 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 665 (time 1769056404.96 total 10.01 last 66.42) - open/close 1278 (time 1769056414.97 total 20.02 last 61.26) - open/close 1952 (time 1769056424.97 total 30.03 last 67.36) - open/close 2605 (time 1769056435.00 total 40.05 last 65.14) - open/close 3297 (time 1769056445.00 total 50.06 last 69.16) - open/close 3935 (time 1769056455.01 total 60.07 last 63.74) - open/close 4578 (time 1769056465.02 total 70.07 last 64.27) - open/close 5286 (time 1769056475.02 total 80.08 last 70.75) - open/close 5955 (time 1769056485.03 total 90.08 last 66.86) - open/close 6667 (time 1769056495.03 total 100.09 last 71.19) - open/close 7249 (time 1769056505.04 total 110.10 last 58.14) - open/close 7815 (time 1769056515.05 total 120.10 last 56.57) - open/close 8447 (time 1769056525.05 total 130.11 last 63.16) - open/close 9086 (time 1769056535.06 total 140.11 last 63.88) - open/close 9782 (time 1769056545.07 total 150.12 last 69.54) total: 10000 open/close in 152.82 seconds: 65.44 ops/second - unlinked 0 (time 1769056552 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 728 (time 1769056660.47 total 10.00 last 72.77) - open/close 1502 (time 1769056670.47 total 20.01 last 77.39) - open/close 2193 (time 1769056680.48 total 30.01 last 69.06) - open/close 2811 (time 1769056690.50 total 40.03 last 61.69) - open/close 3360 (time 1769056700.51 total 50.04 last 54.82) - open/close 4057 (time 1769056710.52 total 60.05 last 69.64) - open/close 4723 (time 1769056720.52 total 70.06 last 66.57) - open/close 5415 (time 1769056730.53 total 80.06 last 69.18) - open/close 6164 (time 1769056740.53 total 90.06 last 74.89) - open/close 6910 (time 1769056750.53 total 100.07 last 74.57) - open/close 7579 (time 1769056760.53 total 110.07 last 66.89) - open/close 8311 (time 1769056770.54 total 120.08 last 73.15) - open/close 8952 (time 1769056780.55 total 130.08 last 64.05) - open/close 9542 (time 1769056790.56 total 140.10 last 58.91) total: 10000 open/close in 148.30 seconds: 67.43 ops/second - unlinked 0 (time 1769056803 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 709 (time 1769056895.05 total 10.00 last 70.90) - open/close 1505 (time 1769056905.07 total 20.02 last 79.46) - open/close 2250 (time 1769056915.07 total 30.02 last 74.49) - open/close 2975 (time 1769056925.07 total 40.02 last 72.46) - open/close 3875 (time 1769056935.08 total 50.03 last 89.95) - open/close 4868 (time 1769056945.08 total 60.03 last 99.28) - open/close 5801 (time 1769056955.09 total 70.04 last 93.19) - open/close 6640 (time 1769056965.11 total 80.06 last 83.75) - open/close 7438 (time 1769056975.12 total 90.07 last 79.73) - open/close 8241 (time 1769056985.13 total 100.08 last 80.20) - open/close 9136 (time 1769056995.13 total 110.09 last 89.47) total: 10000 open/close in 118.71 seconds: 84.24 ops/second - unlinked 0 (time 1769057006 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 884 (time 1769057087.54 total 10.01 last 88.31) - open/close 1536 (time 1769057097.56 total 20.02 last 65.12) - open/close 2332 (time 1769057107.56 total 30.03 last 79.56) - open/close 3128 (time 1769057117.57 total 40.04 last 79.53) - open/close 3930 (time 1769057127.58 total 50.04 last 80.16) - open/close 4643 (time 1769057137.58 total 60.04 last 71.30) - open/close 5393 (time 1769057147.58 total 70.05 last 74.94) - open/close 6216 (time 1769057157.59 total 80.05 last 82.26) - open/close 7071 (time 1769057167.59 total 90.06 last 85.45) - open/close 8039 (time 1769057177.60 total 100.07 last 96.73) - open/close 8922 (time 1769057187.61 total 110.07 last 88.26) - open/close 9845 (time 1769057197.61 total 120.08 last 92.26) total: 10000 open/close in 121.63 seconds: 82.21 ops/second - unlinked 0 (time 1769057202 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 954 (time 1769057277.99 total 10.01 last 95.33) - open/close 1840 (time 1769057288.00 total 20.02 last 88.51) - open/close 2864 (time 1769057298.00 total 30.02 last 102.40) - open/close 3629 (time 1769057308.02 total 40.03 last 76.41) - open/close 4447 (time 1769057318.02 total 50.03 last 81.80) - open/close 5224 (time 1769057328.02 total 60.04 last 77.64) - open/close 6040 (time 1769057338.03 total 70.04 last 81.56) - open/close 6803 (time 1769057348.04 total 80.05 last 76.22) - open/close 7738 (time 1769057358.04 total 90.06 last 93.48) - open/close 8775 (time 1769057368.07 total 100.08 last 103.46) - open/close 9796 (time 1769057378.07 total 110.09 last 102.03) total: 10000 open/close in 112.01 seconds: 89.28 ops/second - unlinked 0 (time 1769057382 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1138 (time 1769057447.49 total 10.00 last 113.79) - open/close 2262 (time 1769057457.49 total 20.01 last 112.30) - open/close 3516 (time 1769057467.51 total 30.02 last 125.22) - open/close 4748 (time 1769057477.51 total 40.03 last 123.16) - open/close 5866 (time 1769057487.52 total 50.03 last 111.75) - open/close 6842 (time 1769057497.53 total 60.04 last 97.50) - open/close 7814 (time 1769057507.53 total 70.04 last 97.18) - open/close 8694 (time 1769057517.54 total 80.05 last 87.94) - open/close 9191 (time 1769057527.55 total 90.07 last 49.61) - open/close 9615 (time 1769057537.56 total 100.08 last 42.37) total: 10000 open/close in 105.80 seconds: 94.52 ops/second - unlinked 0 (time 1769057549 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.127@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.127@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.127@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2