== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 08:53:07 (1761310387) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) oleg306-server: oleg306-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 1505 (time 1761310459.86 total 10.01 last 150.42) - open/close 3677 (time 1761310469.87 total 20.01 last 217.09) - open/close 6095 (time 1761310479.87 total 30.01 last 241.79) - open/close 8967 (time 1761310489.87 total 40.01 last 287.11) total: 10000 open/close in 46.24 seconds: 216.25 ops/second - unlinked 0 (time 1761310497 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1095 (time 1761310553.33 total 10.00 last 109.46) - open/close 2099 (time 1761310563.33 total 20.01 last 100.36) - open/close 3097 (time 1761310573.34 total 30.02 last 99.67) - open/close 4135 (time 1761310583.35 total 40.03 last 103.71) - open/close 5281 (time 1761310593.36 total 50.04 last 114.51) - open/close 6411 (time 1761310603.36 total 60.04 last 112.94) - open/close 7677 (time 1761310613.37 total 70.05 last 126.47) - open/close 9075 (time 1761310623.38 total 80.06 last 139.71) total: 10000 open/close in 84.98 seconds: 117.67 ops/second - unlinked 0 (time 1761310630 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 2662 (time 1761310672.70 total 10.00 last 266.17) - open/close 5285 (time 1761310682.70 total 20.00 last 262.26) - open/close 7861 (time 1761310692.70 total 30.00 last 257.58) total: 10000 open/close in 39.53 seconds: 252.99 ops/second - unlinked 0 (time 1761310703 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2009 (time 1761310741.11 total 10.00 last 200.87) - open/close 4233 (time 1761310751.11 total 20.00 last 222.35) - open/close 6900 (time 1761310761.11 total 30.01 last 266.64) - open/close 9696 (time 1761310771.11 total 40.01 last 279.58) total: 10000 open/close in 41.30 seconds: 242.13 ops/second - unlinked 0 (time 1761310774 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 931 (time 1761310813.13 total 10.00 last 93.09) - open/close 1785 (time 1761310823.14 total 20.01 last 85.36) - open/close 2551 (time 1761310833.15 total 30.02 last 76.52) - open/close 3405 (time 1761310843.15 total 40.02 last 85.36) - open/close 4083 (time 1761310853.15 total 50.02 last 67.77) - open/close 4932 (time 1761310863.16 total 60.03 last 84.83) - open/close 5842 (time 1761310873.17 total 70.04 last 90.96) - open/close 6884 (time 1761310883.17 total 80.04 last 104.19) - open/close 7935 (time 1761310893.17 total 90.04 last 105.05) - open/close 8915 (time 1761310903.17 total 100.04 last 97.98) - open/close 9981 (time 1761310913.18 total 110.05 last 106.53) total: 10000 open/close in 110.18 seconds: 90.76 ops/second - unlinked 0 (time 1761310915 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 738 (time 1761310981.71 total 10.02 last 73.66) - open/close 1557 (time 1761310991.71 total 20.02 last 81.89) - open/close 2441 (time 1761311001.71 total 30.02 last 88.36) - open/close 3350 (time 1761311011.73 total 40.04 last 90.79) - open/close 4296 (time 1761311021.73 total 50.04 last 94.59) - open/close 5202 (time 1761311031.74 total 60.05 last 90.48) - open/close 6123 (time 1761311041.75 total 70.06 last 91.98) - open/close 7098 (time 1761311051.76 total 80.07 last 97.46) - open/close 8125 (time 1761311061.76 total 90.07 last 102.65) - open/close 9344 (time 1761311071.77 total 100.08 last 121.81) total: 10000 open/close in 106.71 seconds: 93.71 ops/second - unlinked 0 (time 1761311080 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1704 (time 1761311132.50 total 10.00 last 170.39) - open/close 3705 (time 1761311142.51 total 20.00 last 200.03) - open/close 5335 (time 1761311152.51 total 30.01 last 162.93) - open/close 6991 (time 1761311162.52 total 40.01 last 165.54) - open/close 8433 (time 1761311172.52 total 50.01 last 144.15) total: 10000 open/close in 59.61 seconds: 167.75 ops/second - unlinked 0 (time 1761311184 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1386 (time 1761311228.26 total 10.02 last 138.35) - open/close 2662 (time 1761311238.26 total 20.02 last 127.53) - open/close 3805 (time 1761311248.27 total 30.03 last 114.24) - open/close 5142 (time 1761311258.27 total 40.03 last 133.70) - open/close 6484 (time 1761311268.27 total 50.03 last 134.18) - open/close 8035 (time 1761311278.27 total 60.03 last 155.04) - open/close 9300 (time 1761311288.28 total 70.04 last 126.39) - open/close 9972 (time 1761311298.30 total 80.06 last 67.11) total: 10000 open/close in 80.64 seconds: 124.01 ops/second - unlinked 0 (time 1761311304 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 880 (time 1761311383.11 total 10.00 last 87.98) - open/close 1887 (time 1761311393.11 total 20.01 last 100.64) - open/close 2601 (time 1761311403.13 total 30.03 last 71.27) - open/close 3181 (time 1761311413.14 total 40.04 last 57.95) - open/close 3692 (time 1761311423.15 total 50.05 last 51.04) - open/close 4160 (time 1761311433.16 total 60.06 last 46.75) - open/close 4697 (time 1761311443.17 total 70.06 last 53.67) - open/close 5378 (time 1761311453.17 total 80.07 last 68.07) - open/close 6193 (time 1761311463.18 total 90.08 last 81.44) - open/close 6987 (time 1761311473.18 total 100.08 last 79.39) - open/close 7846 (time 1761311483.19 total 110.09 last 85.83) - open/close 8650 (time 1761311493.20 total 120.10 last 80.32) - open/close 9518 (time 1761311503.23 total 130.12 last 86.56) total: 10000 open/close in 135.90 seconds: 73.58 ops/second - unlinked 0 (time 1761311512 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 764 (time 1761311595.02 total 10.01 last 76.29) - open/close 1534 (time 1761311605.03 total 20.03 last 76.89) - open/close 2275 (time 1761311615.05 total 30.05 last 73.97) - open/close 2967 (time 1761311625.06 total 40.05 last 69.16) - open/close 3768 (time 1761311635.06 total 50.06 last 80.05) - open/close 4580 (time 1761311645.07 total 60.07 last 81.13) - open/close 5379 (time 1761311655.07 total 70.07 last 79.88) - open/close 6128 (time 1761311665.07 total 80.07 last 74.90) - open/close 6952 (time 1761311675.08 total 90.07 last 82.37) - open/close 7742 (time 1761311685.08 total 100.07 last 79.00) - open/close 8508 (time 1761311695.08 total 110.08 last 76.57) - open/close 9261 (time 1761311705.10 total 120.10 last 75.16) total: 10000 open/close in 129.04 seconds: 77.49 ops/second - unlinked 0 (time 1761311717 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 878 (time 1761311805.64 total 10.01 last 87.75) - open/close 1757 (time 1761311815.64 total 20.01 last 87.85) - open/close 2674 (time 1761311825.66 total 30.03 last 91.56) - open/close 3531 (time 1761311835.66 total 40.03 last 85.66) - open/close 4411 (time 1761311845.67 total 50.04 last 87.92) - open/close 5218 (time 1761311855.69 total 60.06 last 80.55) - open/close 6003 (time 1761311865.70 total 70.07 last 78.38) - open/close 6909 (time 1761311875.71 total 80.08 last 90.56) - open/close 7821 (time 1761311885.71 total 90.08 last 91.17) - open/close 8675 (time 1761311895.72 total 100.09 last 85.33) - open/close 9535 (time 1761311905.72 total 110.09 last 85.98) total: 10000 open/close in 115.85 seconds: 86.31 ops/second - unlinked 0 (time 1761311915 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4352 377216 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322737 s, 3.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg306-server Stopping /mnt/lustre-ost1 (opts:) on oleg306-server reboot facets: ost1 Failover ost1 to oleg306-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.10 seconds: 47.64 ops/second umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg306-server: LNET ready to unload oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.