== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:57:55 (1761238675) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) oleg438-server: oleg438-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg438-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 698 (time 1761238789.48 total 10.01 last 69.74) - open/close 1416 (time 1761238799.50 total 20.02 last 71.70) - open/close 2138 (time 1761238809.51 total 30.04 last 72.10) - open/close 2883 (time 1761238819.51 total 40.04 last 74.48) - open/close 3606 (time 1761238829.52 total 50.04 last 72.28) - open/close 4299 (time 1761238839.52 total 60.05 last 69.25) - open/close 5016 (time 1761238849.53 total 70.06 last 71.62) - open/close 5736 (time 1761238859.55 total 80.07 last 71.90) - open/close 6531 (time 1761238869.56 total 90.08 last 79.43) - open/close 7250 (time 1761238879.56 total 100.09 last 71.88) - open/close 7941 (time 1761238889.56 total 110.09 last 69.09) - open/close 8707 (time 1761238899.57 total 120.09 last 76.57) - open/close 9461 (time 1761238909.58 total 130.10 last 75.31) total: 10000 open/close in 138.29 seconds: 72.31 ops/second - unlinked 0 (time 1761238922 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 682 (time 1761239008.82 total 10.02 last 68.07) - open/close 1432 (time 1761239018.83 total 20.02 last 74.97) - open/close 2171 (time 1761239028.84 total 30.03 last 73.84) - open/close 2940 (time 1761239038.84 total 40.04 last 76.84) - open/close 3757 (time 1761239048.85 total 50.05 last 81.62) - open/close 4585 (time 1761239058.87 total 60.06 last 82.68) - open/close 5332 (time 1761239068.87 total 70.07 last 74.67) - open/close 6085 (time 1761239078.88 total 80.08 last 75.21) - open/close 6923 (time 1761239088.89 total 90.08 last 83.76) - open/close 7785 (time 1761239098.89 total 100.09 last 86.16) - open/close 8715 (time 1761239108.90 total 110.10 last 92.92) - open/close 9545 (time 1761239118.90 total 120.10 last 82.98) total: 10000 open/close in 124.61 seconds: 80.25 ops/second - unlinked 0 (time 1761239125 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 799 (time 1761239201.27 total 10.02 last 79.76) - open/close 1554 (time 1761239211.28 total 20.03 last 75.44) - open/close 2440 (time 1761239221.28 total 30.03 last 88.53) - open/close 3321 (time 1761239231.29 total 40.04 last 88.04) - open/close 4165 (time 1761239241.29 total 50.04 last 84.38) - open/close 4991 (time 1761239251.30 total 60.05 last 82.58) - open/close 5862 (time 1761239261.30 total 70.05 last 87.05) - open/close 6740 (time 1761239271.31 total 80.06 last 87.75) - open/close 7534 (time 1761239281.32 total 90.07 last 79.32) - open/close 8374 (time 1761239291.32 total 100.07 last 83.99) - open/close 9277 (time 1761239301.33 total 110.07 last 90.25) total: 10000 open/close in 119.49 seconds: 83.69 ops/second - unlinked 0 (time 1761239313 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 930 (time 1761239393.40 total 10.01 last 92.91) - open/close 1691 (time 1761239403.40 total 20.02 last 76.04) - open/close 2637 (time 1761239413.41 total 30.02 last 94.58) - open/close 3841 (time 1761239423.42 total 40.03 last 120.24) - open/close 4990 (time 1761239433.43 total 50.04 last 114.82) - open/close 6003 (time 1761239443.43 total 60.05 last 101.24) - open/close 6927 (time 1761239453.44 total 70.05 last 92.34) - open/close 8145 (time 1761239463.44 total 80.06 last 121.77) - open/close 9402 (time 1761239473.45 total 90.07 last 125.58) total: 10000 open/close in 96.83 seconds: 103.27 ops/second - unlinked 0 (time 1761239482 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 939 (time 1761239563.07 total 10.00 last 93.89) - open/close 1806 (time 1761239573.08 total 20.01 last 86.65) - open/close 2729 (time 1761239583.08 total 30.01 last 92.27) - open/close 3737 (time 1761239593.10 total 40.03 last 100.63) - open/close 4532 (time 1761239603.11 total 50.04 last 79.42) - open/close 5355 (time 1761239613.12 total 60.05 last 82.23) - open/close 6257 (time 1761239623.13 total 70.06 last 90.10) - open/close 6982 (time 1761239633.14 total 80.06 last 72.46) - open/close 7768 (time 1761239643.15 total 90.08 last 78.47) - open/close 8509 (time 1761239653.16 total 100.08 last 74.08) - open/close 9331 (time 1761239663.16 total 110.09 last 82.17) total: 10000 open/close in 119.15 seconds: 83.93 ops/second - unlinked 0 (time 1761239675 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 767 (time 1761239756.69 total 10.02 last 76.58) - open/close 1576 (time 1761239766.71 total 20.03 last 80.81) - open/close 2417 (time 1761239776.71 total 30.03 last 84.09) - open/close 3311 (time 1761239786.72 total 40.04 last 89.29) - open/close 4224 (time 1761239796.73 total 50.05 last 91.21) - open/close 5098 (time 1761239806.73 total 60.05 last 87.35) - open/close 5958 (time 1761239816.74 total 70.06 last 85.96) - open/close 6859 (time 1761239826.74 total 80.07 last 90.05) - open/close 7715 (time 1761239836.75 total 90.07 last 85.58) - open/close 8615 (time 1761239846.76 total 100.08 last 89.87) - open/close 9567 (time 1761239856.76 total 110.08 last 95.17) total: 10000 open/close in 114.39 seconds: 87.42 ops/second - unlinked 0 (time 1761239863 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1307 (time 1761239925.11 total 10.01 last 130.63) - open/close 2336 (time 1761239935.11 total 20.01 last 102.89) - open/close 3206 (time 1761239945.13 total 30.02 last 86.90) - open/close 3978 (time 1761239955.14 total 40.03 last 77.11) - open/close 4700 (time 1761239965.15 total 50.04 last 72.14) - open/close 5487 (time 1761239975.16 total 60.05 last 78.63) - open/close 6446 (time 1761239985.17 total 70.06 last 95.73) - open/close 7292 (time 1761239995.17 total 80.06 last 84.60) - open/close 8093 (time 1761240005.18 total 90.07 last 80.06) - open/close 9015 (time 1761240015.18 total 100.08 last 92.14) total: 10000 open/close in 109.64 seconds: 91.21 ops/second - unlinked 0 (time 1761240028 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 774 (time 1761240116.72 total 10.00 last 77.36) - open/close 1479 (time 1761240126.72 total 20.01 last 70.45) - open/close 2179 (time 1761240136.73 total 30.02 last 69.94) - open/close 2908 (time 1761240146.74 total 40.03 last 72.84) - open/close 3634 (time 1761240156.75 total 50.04 last 72.51) - open/close 4387 (time 1761240166.77 total 60.06 last 75.18) - open/close 5227 (time 1761240176.78 total 70.07 last 83.91) - open/close 6008 (time 1761240186.79 total 80.08 last 77.98) - open/close 6886 (time 1761240196.80 total 90.08 last 87.79) - open/close 7703 (time 1761240206.80 total 100.09 last 81.66) - open/close 8499 (time 1761240216.81 total 110.09 last 79.57) - open/close 9236 (time 1761240226.81 total 120.10 last 73.66) - open/close 9950 (time 1761240236.83 total 130.11 last 71.29) total: 10000 open/close in 130.75 seconds: 76.48 ops/second - unlinked 0 (time 1761240240 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 753 (time 1761240329.93 total 10.00 last 75.29) - open/close 1521 (time 1761240339.94 total 20.02 last 76.68) - open/close 2239 (time 1761240349.96 total 30.03 last 71.72) - open/close 3118 (time 1761240359.96 total 40.03 last 87.89) - open/close 4019 (time 1761240369.97 total 50.04 last 90.02) - open/close 4806 (time 1761240379.97 total 60.04 last 78.67) - open/close 5562 (time 1761240389.98 total 70.05 last 75.54) - open/close 6374 (time 1761240399.98 total 80.06 last 81.14) - open/close 7218 (time 1761240409.99 total 90.06 last 84.38) - open/close 7938 (time 1761240419.99 total 100.06 last 72.00) - open/close 8584 (time 1761240429.99 total 110.06 last 64.58) - open/close 9212 (time 1761240440.00 total 120.07 last 62.77) - open/close 9878 (time 1761240450.00 total 130.07 last 66.59) total: 10000 open/close in 131.63 seconds: 75.97 ops/second - unlinked 0 (time 1761240454 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 620 (time 1761240552.80 total 10.01 last 61.92) - open/close 1255 (time 1761240562.81 total 20.02 last 63.43) - open/close 1898 (time 1761240572.81 total 30.03 last 64.27) - open/close 2525 (time 1761240582.81 total 40.03 last 62.70) - open/close 3196 (time 1761240592.82 total 50.04 last 67.04) - open/close 3811 (time 1761240602.83 total 60.04 last 61.45) - open/close 4485 (time 1761240612.85 total 70.06 last 67.29) - open/close 5174 (time 1761240622.86 total 80.07 last 68.83) - open/close 5906 (time 1761240632.86 total 90.08 last 73.14) - open/close 6595 (time 1761240642.88 total 100.09 last 68.81) - open/close 7243 (time 1761240652.89 total 110.10 last 64.74) - open/close 7910 (time 1761240662.91 total 120.13 last 66.54) - open/close 8609 (time 1761240672.92 total 130.13 last 69.85) - open/close 9348 (time 1761240682.92 total 140.14 last 73.87) total: 10000 open/close in 149.36 seconds: 66.95 ops/second - unlinked 0 (time 1761240695 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 622 (time 1761240794.49 total 10.01 last 62.14) - open/close 1347 (time 1761240804.51 total 20.03 last 72.32) - open/close 2103 (time 1761240814.52 total 30.04 last 75.53) - open/close 2877 (time 1761240824.53 total 40.05 last 77.35) - open/close 3562 (time 1761240834.53 total 50.05 last 68.47) - open/close 4263 (time 1761240844.53 total 60.05 last 70.10) - open/close 4992 (time 1761240854.54 total 70.06 last 72.84) - open/close 5700 (time 1761240864.55 total 80.07 last 70.73) - open/close 6451 (time 1761240874.56 total 90.08 last 75.03) - open/close 7143 (time 1761240884.57 total 100.09 last 69.14) - open/close 7842 (time 1761240894.58 total 110.10 last 69.86) - open/close 8562 (time 1761240904.58 total 120.10 last 71.98) - open/close 9318 (time 1761240914.58 total 130.10 last 75.57) total: 10000 open/close in 140.08 seconds: 71.39 ops/second - unlinked 0 (time 1761240927 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513588 s, 2.0 MB/s fail_loc=0x80000204 Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server reboot facets: ost1 Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761241092.34 total 36.95 last 0.00) total: 100 open/close in 38.92 seconds: 2.57 ops/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg438-server: LNET ready to unload oleg438-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.