== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 04:23:43 (1761121423) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) oleg124-server: oleg124-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg124-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock oleg124-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 3568 (time 1761121480.96 total 10.00 last 356.67) - open/close 6497 (time 1761121490.96 total 20.00 last 292.86) - open/close 9802 (time 1761121500.97 total 30.01 last 330.41) total: 10000 open/close in 30.60 seconds: 326.75 ops/second - unlinked 0 (time 1761121503 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3036 (time 1761121532.77 total 10.00 last 303.58) - open/close 5613 (time 1761121542.77 total 20.00 last 257.70) - open/close 7906 (time 1761121552.77 total 30.00 last 229.29) total: 10000 open/close in 38.94 seconds: 256.78 ops/second - unlinked 0 (time 1761121563 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1203 (time 1761121602.83 total 10.01 last 120.22) - open/close 2381 (time 1761121612.84 total 20.01 last 117.71) - open/close 3600 (time 1761121622.84 total 30.02 last 121.86) - open/close 5247 (time 1761121632.86 total 40.03 last 164.45) - open/close 7096 (time 1761121642.87 total 50.04 last 184.72) - open/close 8631 (time 1761121652.87 total 60.05 last 153.45) total: 10000 open/close in 67.61 seconds: 147.90 ops/second - unlinked 0 (time 1761121662 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2374 (time 1761121699.18 total 10.00 last 237.35) - open/close 5015 (time 1761121709.18 total 20.00 last 264.09) - open/close 7337 (time 1761121719.19 total 30.01 last 232.12) - open/close 9748 (time 1761121729.19 total 40.01 last 241.07) total: 10000 open/close in 41.11 seconds: 243.26 ops/second - unlinked 0 (time 1761121732 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2517 (time 1761121769.10 total 10.00 last 251.69) - open/close 5292 (time 1761121779.10 total 20.00 last 277.43) - open/close 7765 (time 1761121789.10 total 30.00 last 247.25) - open/close 9826 (time 1761121799.11 total 40.01 last 205.96) total: 10000 open/close in 40.83 seconds: 244.93 ops/second - unlinked 0 (time 1761121801 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2128 (time 1761121839.74 total 10.00 last 212.79) - open/close 4413 (time 1761121849.75 total 20.00 last 228.44) - open/close 6346 (time 1761121859.75 total 30.01 last 193.19) - open/close 8732 (time 1761121869.75 total 40.01 last 238.53) total: 10000 open/close in 45.57 seconds: 219.43 ops/second - unlinked 0 (time 1761121877 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2173 (time 1761121909.33 total 10.01 last 217.19) - open/close 4719 (time 1761121919.33 total 20.01 last 254.53) - open/close 6436 (time 1761121929.33 total 30.01 last 171.66) - open/close 8918 (time 1761121939.34 total 40.02 last 248.08) total: 10000 open/close in 44.86 seconds: 222.92 ops/second - unlinked 0 (time 1761121946 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2342 (time 1761121983.64 total 10.00 last 234.15) - open/close 4463 (time 1761121993.64 total 20.01 last 211.97) - open/close 6609 (time 1761122003.64 total 30.01 last 214.58) - open/close 9186 (time 1761122013.65 total 40.01 last 257.62) total: 10000 open/close in 43.17 seconds: 231.65 ops/second - unlinked 0 (time 1761122018 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 805 (time 1761122113.85 total 10.01 last 80.41) - open/close 1670 (time 1761122123.86 total 20.02 last 86.45) - open/close 2674 (time 1761122133.86 total 30.02 last 100.37) - open/close 3637 (time 1761122143.86 total 40.02 last 96.29) - open/close 4458 (time 1761122153.87 total 50.03 last 82.01) - open/close 5358 (time 1761122163.87 total 60.04 last 89.96) - open/close 6121 (time 1761122173.88 total 70.04 last 76.23) - open/close 6991 (time 1761122183.89 total 80.05 last 86.99) - open/close 7833 (time 1761122193.89 total 90.05 last 84.17) - open/close 8724 (time 1761122203.89 total 100.05 last 89.09) - open/close 9727 (time 1761122213.89 total 110.05 last 100.26) total: 10000 open/close in 112.48 seconds: 88.90 ops/second - unlinked 0 (time 1761122218 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1018 (time 1761122293.01 total 10.00 last 101.79) - open/close 1886 (time 1761122303.02 total 20.01 last 86.72) - open/close 2715 (time 1761122313.02 total 30.01 last 82.86) - open/close 3529 (time 1761122323.03 total 40.02 last 81.34) - open/close 4305 (time 1761122333.04 total 50.03 last 77.54) - open/close 4836 (time 1761122343.09 total 60.08 last 52.84) - open/close 5241 (time 1761122353.10 total 70.09 last 40.46) - open/close 5664 (time 1761122363.12 total 80.11 last 42.23) - open/close 6034 (time 1761122373.14 total 90.13 last 36.93) - open/close 6447 (time 1761122383.15 total 100.14 last 41.25) - open/close 6855 (time 1761122393.17 total 110.17 last 40.69) - open/close 7412 (time 1761122403.20 total 120.19 last 55.57) - open/close 7989 (time 1761122413.25 total 130.25 last 57.38) - open/close 8587 (time 1761122423.26 total 140.25 last 59.74) - open/close 9210 (time 1761122433.26 total 150.26 last 62.29) - open/close 9851 (time 1761122443.28 total 160.27 last 64.02) total: 10000 open/close in 162.36 seconds: 61.59 ops/second - unlinked 0 (time 1761122449 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 630 (time 1761122556.46 total 10.00 last 62.99) - open/close 1268 (time 1761122566.46 total 20.01 last 63.76) - open/close 2038 (time 1761122576.46 total 30.01 last 77.00) - open/close 2743 (time 1761122586.48 total 40.02 last 70.41) - open/close 3389 (time 1761122596.48 total 50.03 last 64.57) - open/close 4056 (time 1761122606.48 total 60.03 last 66.68) - open/close 4740 (time 1761122616.48 total 70.03 last 68.40) - open/close 5411 (time 1761122626.49 total 80.03 last 67.08) - open/close 6032 (time 1761122636.51 total 90.06 last 61.95) - open/close 6647 (time 1761122646.52 total 100.07 last 61.42) - open/close 7276 (time 1761122656.54 total 110.08 last 62.83) - open/close 7930 (time 1761122666.54 total 120.09 last 65.38) - open/close 8542 (time 1761122676.56 total 130.11 last 61.08) - open/close 9130 (time 1761122686.57 total 140.12 last 58.72) - open/close 9743 (time 1761122696.58 total 150.13 last 61.24) total: 10000 open/close in 154.56 seconds: 64.70 ops/second - unlinked 0 (time 1761122705 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389467 s, 2.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg124-server Stopping /mnt/lustre-ost1 (opts:) on oleg124-server reboot facets: ost1 Failover ost1 to oleg124-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761122872.73 total 35.96 last 0.00) total: 100 open/close in 38.83 seconds: 2.58 ops/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg124-server: LNET ready to unload oleg124-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.