== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 08:56:57 (1761310617) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) oleg205-server: oleg205-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg205-server: detected 4 online CPUs by sysfs oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg205-server: detected 4 online CPUs by sysfs oleg205-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg205-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 1178 (time 1761310691.18 total 10.01 last 117.73) - open/close 2399 (time 1761310701.19 total 20.01 last 122.02) - open/close 3632 (time 1761310711.19 total 30.02 last 123.26) - open/close 5211 (time 1761310721.19 total 40.02 last 157.88) - open/close 6547 (time 1761310731.19 total 50.02 last 133.57) - open/close 7748 (time 1761310741.20 total 60.02 last 120.07) - open/close 8895 (time 1761310751.20 total 70.02 last 114.69) total: 10000 open/close in 78.37 seconds: 127.60 ops/second - unlinked 0 (time 1761310761 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 640 (time 1761310832.72 total 10.01 last 63.92) - open/close 1335 (time 1761310842.73 total 20.02 last 69.44) - open/close 2130 (time 1761310852.73 total 30.02 last 79.50) - open/close 2903 (time 1761310862.73 total 40.02 last 77.29) - open/close 3712 (time 1761310872.74 total 50.03 last 80.88) - open/close 4560 (time 1761310882.75 total 60.04 last 84.71) - open/close 5429 (time 1761310892.75 total 70.04 last 86.85) - open/close 6244 (time 1761310902.76 total 80.05 last 81.41) - open/close 7084 (time 1761310912.78 total 90.07 last 83.88) - open/close 7953 (time 1761310922.78 total 100.07 last 86.87) - open/close 8818 (time 1761310932.79 total 110.08 last 86.43) - open/close 9673 (time 1761310942.80 total 120.08 last 85.44) total: 10000 open/close in 124.01 seconds: 80.64 ops/second - unlinked 0 (time 1761310949 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 842 (time 1761311026.12 total 10.00 last 84.17) - open/close 1745 (time 1761311036.13 total 20.01 last 90.20) - open/close 2607 (time 1761311046.14 total 30.02 last 86.16) - open/close 3562 (time 1761311056.15 total 40.03 last 95.42) - open/close 4514 (time 1761311066.16 total 50.04 last 95.05) - open/close 5488 (time 1761311076.17 total 60.05 last 97.35) - open/close 6531 (time 1761311086.17 total 70.06 last 104.23) - open/close 7463 (time 1761311096.18 total 80.06 last 93.12) - open/close 8502 (time 1761311106.19 total 90.07 last 103.85) - open/close 9480 (time 1761311116.19 total 100.07 last 97.79) total: 10000 open/close in 105.98 seconds: 94.35 ops/second - unlinked 0 (time 1761311124 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 932 (time 1761311188.98 total 10.00 last 93.18) - open/close 1944 (time 1761311198.99 total 20.01 last 101.15) - open/close 3000 (time 1761311208.99 total 30.01 last 105.54) - open/close 4139 (time 1761311218.99 total 40.01 last 113.89) - open/close 5294 (time 1761311229.00 total 50.02 last 115.48) - open/close 6472 (time 1761311239.00 total 60.02 last 117.71) - open/close 7628 (time 1761311249.01 total 70.03 last 115.55) - open/close 9008 (time 1761311259.01 total 80.03 last 137.99) total: 10000 open/close in 86.38 seconds: 115.76 ops/second - unlinked 0 (time 1761311267 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 902 (time 1761311343.49 total 10.00 last 90.16) - open/close 1828 (time 1761311353.49 total 20.01 last 92.57) - open/close 2746 (time 1761311363.49 total 30.01 last 91.79) - open/close 3858 (time 1761311373.50 total 40.01 last 111.16) - open/close 4871 (time 1761311383.50 total 50.01 last 101.28) - open/close 5840 (time 1761311393.52 total 60.03 last 96.73) - open/close 6616 (time 1761311403.54 total 70.05 last 77.44) - open/close 7234 (time 1761311413.55 total 80.06 last 61.74) - open/close 7858 (time 1761311423.55 total 90.07 last 62.36) - open/close 8572 (time 1761311433.56 total 100.07 last 71.35) - open/close 9476 (time 1761311443.57 total 110.09 last 90.29) total: 10000 open/close in 116.12 seconds: 86.12 ops/second - unlinked 0 (time 1761311452 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 781 (time 1761311536.31 total 10.01 last 78.01) - open/close 1743 (time 1761311546.32 total 20.02 last 96.15) - open/close 2628 (time 1761311556.32 total 30.02 last 88.43) - open/close 3499 (time 1761311566.33 total 40.03 last 87.02) - open/close 4350 (time 1761311576.34 total 50.04 last 85.02) - open/close 5168 (time 1761311586.35 total 60.05 last 81.76) - open/close 5974 (time 1761311596.36 total 70.06 last 80.51) - open/close 6804 (time 1761311606.37 total 80.07 last 82.93) - open/close 7645 (time 1761311616.37 total 90.07 last 84.04) - open/close 8426 (time 1761311626.38 total 100.08 last 78.05) - open/close 9237 (time 1761311636.39 total 110.09 last 81.05) total: 10000 open/close in 119.35 seconds: 83.79 ops/second - unlinked 0 (time 1761311649 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 842 (time 1761311733.65 total 10.02 last 84.04) - open/close 1718 (time 1761311743.66 total 20.03 last 87.52) - open/close 2556 (time 1761311753.66 total 30.03 last 83.75) - open/close 3413 (time 1761311763.67 total 40.04 last 85.65) - open/close 4319 (time 1761311773.68 total 50.05 last 90.54) - open/close 5174 (time 1761311783.68 total 60.05 last 85.47) - open/close 6025 (time 1761311793.68 total 70.05 last 85.09) - open/close 6904 (time 1761311803.69 total 80.06 last 87.84) - open/close 7834 (time 1761311813.69 total 90.06 last 92.93) - open/close 8659 (time 1761311823.70 total 100.07 last 82.48) - open/close 9463 (time 1761311833.71 total 110.08 last 80.27) total: 10000 open/close in 116.30 seconds: 85.98 ops/second - unlinked 0 (time 1761311843 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 824 (time 1761311931.36 total 10.01 last 82.35) - open/close 1657 (time 1761311941.38 total 20.02 last 83.18) - open/close 2479 (time 1761311951.39 total 30.03 last 82.13) - open/close 3333 (time 1761311961.39 total 40.03 last 85.39) - open/close 4188 (time 1761311971.40 total 50.04 last 85.44) - open/close 4973 (time 1761311981.40 total 60.05 last 78.43) - open/close 5756 (time 1761311991.42 total 70.06 last 78.19) - open/close 6582 (time 1761312001.42 total 80.07 last 82.55) - open/close 7399 (time 1761312011.42 total 90.07 last 81.70) - open/close 8236 (time 1761312021.43 total 100.07 last 83.68) - open/close 9038 (time 1761312031.44 total 110.08 last 80.09) - open/close 9780 (time 1761312041.44 total 120.09 last 74.17) total: 10000 open/close in 122.65 seconds: 81.53 ops/second - unlinked 0 (time 1761312046 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 951 (time 1761312124.74 total 10.00 last 95.06) - open/close 1812 (time 1761312134.76 total 20.02 last 85.93) - open/close 2705 (time 1761312144.77 total 30.04 last 89.19) - open/close 3542 (time 1761312154.77 total 40.04 last 83.70) - open/close 4350 (time 1761312164.78 total 50.04 last 80.74) - open/close 5187 (time 1761312174.78 total 60.05 last 83.62) - open/close 6099 (time 1761312184.79 total 70.06 last 91.11) - open/close 6893 (time 1761312194.80 total 80.07 last 79.36) - open/close 7679 (time 1761312204.81 total 90.08 last 78.49) - open/close 8500 (time 1761312214.83 total 100.10 last 81.98) - open/close 9311 (time 1761312224.85 total 110.11 last 80.95) total: 10000 open/close in 117.99 seconds: 84.75 ops/second - unlinked 0 (time 1761312235 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 787 (time 1761312315.68 total 10.00 last 78.67) - open/close 1574 (time 1761312325.69 total 20.02 last 78.59) - open/close 2329 (time 1761312335.70 total 30.02 last 75.46) - open/close 3292 (time 1761312345.71 total 40.03 last 96.21) - open/close 4250 (time 1761312355.71 total 50.03 last 95.78) - open/close 5277 (time 1761312365.72 total 60.04 last 102.64) - open/close 6207 (time 1761312375.72 total 70.04 last 93.00) - open/close 7043 (time 1761312385.72 total 80.04 last 83.59) - open/close 7900 (time 1761312395.72 total 90.05 last 85.66) - open/close 8789 (time 1761312405.73 total 100.05 last 88.84) - open/close 9709 (time 1761312415.74 total 110.06 last 91.93) total: 10000 open/close in 113.69 seconds: 87.96 ops/second - unlinked 0 (time 1761312422 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 893 (time 1761312496.54 total 10.00 last 89.28) - open/close 1795 (time 1761312506.54 total 20.00 last 90.19) - open/close 2683 (time 1761312516.54 total 30.01 last 88.78) - open/close 3574 (time 1761312526.55 total 40.02 last 88.98) - open/close 4452 (time 1761312536.56 total 50.02 last 87.78) - open/close 5310 (time 1761312546.57 total 60.03 last 85.70) - open/close 6189 (time 1761312556.58 total 70.04 last 87.81) - open/close 7011 (time 1761312566.59 total 80.06 last 82.09) - open/close 7831 (time 1761312576.60 total 90.06 last 81.94) - open/close 8680 (time 1761312586.60 total 100.07 last 84.85) - open/close 9498 (time 1761312596.61 total 110.08 last 81.74) total: 10000 open/close in 116.25 seconds: 86.02 ops/second - unlinked 0 (time 1761312606 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301521 s, 3.5 MB/s fail_loc=0x80000204 Failing ost1 on oleg205-server Stopping /mnt/lustre-ost1 (opts:) on oleg205-server reboot facets: ost1 Failover ost1 to oleg205-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761312752.90 total 37.48 last 0.00) total: 100 open/close in 39.66 seconds: 2.52 ops/second umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg205-server: LNET ready to unload oleg205-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.