== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:14:09 (1761311649) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) oleg122-server: oleg122-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy fail_loc=0x00001e0 start mds service on oleg122-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock oleg122-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 753 (time 1761311773.09 total 10.00 last 75.27) - open/close 1511 (time 1761311783.10 total 20.01 last 75.74) - open/close 2248 (time 1761311793.12 total 30.03 last 73.59) - open/close 3017 (time 1761311803.12 total 40.03 last 76.86) - open/close 3772 (time 1761311813.13 total 50.04 last 75.48) - open/close 4481 (time 1761311823.13 total 60.04 last 70.86) - open/close 5232 (time 1761311833.14 total 70.05 last 75.03) - open/close 5910 (time 1761311843.16 total 80.07 last 67.67) - open/close 6637 (time 1761311853.17 total 90.08 last 72.66) - open/close 7338 (time 1761311863.17 total 100.08 last 70.10) - open/close 8060 (time 1761311873.17 total 110.08 last 72.18) - open/close 8801 (time 1761311883.17 total 120.08 last 74.06) - open/close 9524 (time 1761311893.18 total 130.09 last 72.28) total: 10000 open/close in 136.09 seconds: 73.48 ops/second - unlinked 0 (time 1761311902 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 795 (time 1761311995.28 total 10.00 last 79.49) - open/close 1554 (time 1761312005.28 total 20.01 last 75.87) - open/close 2306 (time 1761312015.29 total 30.01 last 75.15) - open/close 3053 (time 1761312025.30 total 40.02 last 74.65) - open/close 3787 (time 1761312035.31 total 50.04 last 73.27) - open/close 4557 (time 1761312045.32 total 60.04 last 76.97) - open/close 5303 (time 1761312055.32 total 70.05 last 74.55) - open/close 6072 (time 1761312065.33 total 80.05 last 76.90) - open/close 6766 (time 1761312075.33 total 90.05 last 69.35) - open/close 7573 (time 1761312085.34 total 100.06 last 80.67) - open/close 8297 (time 1761312095.34 total 110.06 last 72.40) - open/close 9035 (time 1761312105.34 total 120.06 last 73.77) - open/close 9777 (time 1761312115.35 total 130.07 last 74.13) total: 10000 open/close in 133.08 seconds: 75.15 ops/second - unlinked 0 (time 1761312122 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 798 (time 1761312210.53 total 10.01 last 79.76) - open/close 1589 (time 1761312220.54 total 20.02 last 79.01) - open/close 2373 (time 1761312230.55 total 30.02 last 78.35) - open/close 3058 (time 1761312240.55 total 40.03 last 68.47) - open/close 3842 (time 1761312250.57 total 50.05 last 78.26) - open/close 4594 (time 1761312260.58 total 60.05 last 75.14) - open/close 5338 (time 1761312270.58 total 70.06 last 74.36) - open/close 6088 (time 1761312280.60 total 80.08 last 74.86) - open/close 6820 (time 1761312290.61 total 90.08 last 73.15) - open/close 7559 (time 1761312300.62 total 100.09 last 73.83) - open/close 8259 (time 1761312310.63 total 110.10 last 69.94) - open/close 9036 (time 1761312320.64 total 120.11 last 77.62) - open/close 9810 (time 1761312330.64 total 130.12 last 77.36) total: 10000 open/close in 132.40 seconds: 75.53 ops/second - unlinked 0 (time 1761312335 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 795 (time 1761312417.06 total 10.00 last 79.49) - open/close 1597 (time 1761312427.07 total 20.01 last 80.15) - open/close 2378 (time 1761312437.08 total 30.02 last 78.00) - open/close 3199 (time 1761312447.08 total 40.02 last 82.08) - open/close 4029 (time 1761312457.09 total 50.03 last 82.92) - open/close 4903 (time 1761312467.09 total 60.03 last 87.37) - open/close 5767 (time 1761312477.11 total 70.05 last 86.24) - open/close 6557 (time 1761312487.12 total 80.06 last 78.93) - open/close 7375 (time 1761312497.13 total 90.07 last 81.73) - open/close 8206 (time 1761312507.14 total 100.08 last 83.04) - open/close 9055 (time 1761312517.14 total 110.08 last 84.88) - open/close 9889 (time 1761312527.15 total 120.09 last 83.31) total: 10000 open/close in 121.41 seconds: 82.37 ops/second - unlinked 0 (time 1761312531 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1062 (time 1761312606.64 total 10.01 last 106.14) - open/close 2006 (time 1761312616.64 total 20.01 last 94.38) - open/close 2905 (time 1761312626.65 total 30.02 last 89.82) - open/close 3792 (time 1761312636.65 total 40.02 last 88.69) - open/close 4829 (time 1761312646.65 total 50.02 last 103.63) - open/close 5754 (time 1761312656.66 total 60.03 last 92.44) - open/close 6605 (time 1761312666.67 total 70.04 last 85.04) - open/close 7519 (time 1761312676.68 total 80.04 last 91.33) - open/close 8384 (time 1761312686.68 total 90.05 last 86.43) - open/close 9235 (time 1761312696.69 total 100.06 last 85.02) total: 10000 open/close in 108.17 seconds: 92.45 ops/second - unlinked 0 (time 1761312707 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 843 (time 1761312783.88 total 10.01 last 84.25) - open/close 1720 (time 1761312793.88 total 20.01 last 87.62) - open/close 2611 (time 1761312803.89 total 30.03 last 89.01) - open/close 3462 (time 1761312813.91 total 40.04 last 85.00) - open/close 4290 (time 1761312823.91 total 50.04 last 82.77) - open/close 5114 (time 1761312833.93 total 60.06 last 82.25) - open/close 5959 (time 1761312843.93 total 70.06 last 84.49) - open/close 6715 (time 1761312853.93 total 80.06 last 75.59) - open/close 7493 (time 1761312863.94 total 90.07 last 77.72) - open/close 8387 (time 1761312873.95 total 100.08 last 89.36) - open/close 9221 (time 1761312883.95 total 110.08 last 83.35) total: 10000 open/close in 119.49 seconds: 83.69 ops/second - unlinked 0 (time 1761312896 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 905 (time 1761312976.67 total 10.00 last 90.47) - open/close 1775 (time 1761312986.68 total 20.01 last 86.93) - open/close 2641 (time 1761312996.69 total 30.02 last 86.54) - open/close 3502 (time 1761313006.69 total 40.02 last 86.06) - open/close 4388 (time 1761313016.70 total 50.03 last 88.53) - open/close 5273 (time 1761313026.70 total 60.03 last 88.46) - open/close 6107 (time 1761313036.71 total 70.04 last 83.34) - open/close 6945 (time 1761313046.72 total 80.05 last 83.73) - open/close 7796 (time 1761313056.72 total 90.05 last 85.09) - open/close 8633 (time 1761313066.73 total 100.06 last 83.60) - open/close 9468 (time 1761313076.74 total 110.08 last 83.39) total: 10000 open/close in 115.97 seconds: 86.23 ops/second - unlinked 0 (time 1761313085 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 831 (time 1761313171.56 total 10.00 last 83.08) - open/close 1655 (time 1761313181.57 total 20.01 last 82.31) - open/close 2466 (time 1761313191.58 total 30.02 last 81.01) - open/close 3379 (time 1761313201.59 total 40.03 last 91.26) - open/close 4324 (time 1761313211.60 total 50.04 last 94.42) - open/close 5221 (time 1761313221.62 total 60.06 last 89.51) - open/close 6144 (time 1761313231.63 total 70.07 last 92.19) - open/close 7062 (time 1761313241.64 total 80.08 last 91.73) - open/close 7922 (time 1761313251.65 total 90.09 last 85.87) - open/close 8824 (time 1761313261.65 total 100.09 last 90.19) - open/close 9777 (time 1761313271.66 total 110.10 last 95.25) total: 10000 open/close in 112.51 seconds: 88.88 ops/second - unlinked 0 (time 1761313277 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 915 (time 1761313351.56 total 10.00 last 91.49) - open/close 1763 (time 1761313361.56 total 20.01 last 84.76) - open/close 2594 (time 1761313371.56 total 30.01 last 83.08) - open/close 3414 (time 1761313381.57 total 40.01 last 81.95) - open/close 4246 (time 1761313391.57 total 50.02 last 83.18) - open/close 5122 (time 1761313401.57 total 60.02 last 87.58) - open/close 6031 (time 1761313411.58 total 70.03 last 90.84) - open/close 6929 (time 1761313421.59 total 80.04 last 89.68) - open/close 7724 (time 1761313431.60 total 90.05 last 79.42) - open/close 8548 (time 1761313441.61 total 100.05 last 82.37) - open/close 9378 (time 1761313451.62 total 110.06 last 82.93) total: 10000 open/close in 117.42 seconds: 85.16 ops/second - unlinked 0 (time 1761313463 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 904 (time 1761313547.62 total 10.01 last 90.35) - open/close 1803 (time 1761313557.63 total 20.01 last 89.88) - open/close 2747 (time 1761313567.63 total 30.01 last 94.37) - open/close 3639 (time 1761313577.64 total 40.02 last 89.14) - open/close 4569 (time 1761313587.64 total 50.02 last 93.00) - open/close 5396 (time 1761313597.65 total 60.03 last 82.60) - open/close 6272 (time 1761313607.65 total 70.03 last 87.56) - open/close 7142 (time 1761313617.66 total 80.04 last 86.92) - open/close 7989 (time 1761313627.66 total 90.04 last 84.69) - open/close 8844 (time 1761313637.67 total 100.05 last 85.48) - open/close 9666 (time 1761313647.68 total 110.06 last 82.08) total: 10000 open/close in 113.96 seconds: 87.75 ops/second - unlinked 0 (time 1761313654 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1024 (time 1761313722.80 total 10.00 last 102.35) - open/close 1920 (time 1761313732.80 total 20.01 last 89.53) - open/close 3046 (time 1761313742.82 total 30.03 last 112.44) - open/close 4159 (time 1761313752.82 total 40.03 last 111.27) - open/close 5040 (time 1761313762.82 total 50.03 last 88.06) - open/close 5870 (time 1761313772.83 total 60.04 last 82.97) - open/close 6733 (time 1761313782.84 total 70.05 last 86.23) - open/close 7575 (time 1761313792.84 total 80.05 last 84.15) - open/close 8495 (time 1761313802.85 total 90.05 last 91.97) - open/close 9409 (time 1761313812.85 total 100.06 last 91.35) total: 10000 open/close in 106.58 seconds: 93.83 ops/second - unlinked 0 (time 1761313821 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242344 s, 4.3 MB/s fail_loc=0x80000204 Failing ost1 on oleg122-server Stopping /mnt/lustre-ost1 (opts:) on oleg122-server reboot facets: ost1 Failover ost1 to oleg122-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.41 seconds: 41.50 ops/second umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg122-server: LNET ready to unload oleg122-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.