== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:00:53 (1761310853) Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: oleg140-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg140-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 1174 (time 1761310963.28 total 10.00 last 117.35) - open/close 2099 (time 1761310973.28 total 20.01 last 92.45) - open/close 3071 (time 1761310983.29 total 30.01 last 97.15) - open/close 3930 (time 1761310993.29 total 40.02 last 85.85) - open/close 4784 (time 1761311003.30 total 50.02 last 85.36) - open/close 6096 (time 1761311013.30 total 60.03 last 131.17) - open/close 7281 (time 1761311023.30 total 70.03 last 118.45) - open/close 9227 (time 1761311033.30 total 80.03 last 194.59) total: 10000 open/close in 82.49 seconds: 121.22 ops/second - unlinked 0 (time 1761311037 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 3014 (time 1761311077.54 total 10.00 last 301.29) - open/close 5567 (time 1761311087.54 total 20.00 last 255.28) - open/close 8240 (time 1761311097.55 total 30.01 last 267.05) total: 10000 open/close in 36.34 seconds: 275.15 ops/second - unlinked 0 (time 1761311105 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 3406 (time 1761311137.57 total 10.00 last 340.54) - open/close 6486 (time 1761311147.57 total 20.01 last 307.89) - open/close 8555 (time 1761311157.57 total 30.01 last 206.89) total: 10000 open/close in 36.67 seconds: 272.69 ops/second - unlinked 0 (time 1761311165 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2451 (time 1761311197.41 total 10.00 last 245.07) - open/close 5188 (time 1761311207.42 total 20.00 last 273.60) - open/close 7737 (time 1761311217.42 total 30.01 last 254.89) total: 10000 open/close in 39.02 seconds: 256.29 ops/second - unlinked 0 (time 1761311227 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2243 (time 1761311258.10 total 10.00 last 224.28) - open/close 4736 (time 1761311268.10 total 20.00 last 249.25) - open/close 8111 (time 1761311278.10 total 30.00 last 337.48) total: 10000 open/close in 35.46 seconds: 281.98 ops/second - unlinked 0 (time 1761311285 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 833 (time 1761311366.48 total 10.01 last 83.22) - open/close 1691 (time 1761311376.49 total 20.01 last 85.76) - open/close 2660 (time 1761311386.49 total 30.02 last 96.89) - open/close 3538 (time 1761311396.50 total 40.03 last 87.69) - open/close 4226 (time 1761311406.51 total 50.04 last 68.74) - open/close 5040 (time 1761311416.51 total 60.04 last 81.38) - open/close 5818 (time 1761311426.52 total 70.05 last 77.70) - open/close 6574 (time 1761311436.53 total 80.06 last 75.57) - open/close 7334 (time 1761311446.53 total 90.06 last 75.98) - open/close 8219 (time 1761311456.54 total 100.07 last 88.41) - open/close 9043 (time 1761311466.54 total 110.07 last 82.40) - open/close 9882 (time 1761311476.55 total 120.08 last 83.85) total: 10000 open/close in 121.23 seconds: 82.49 ops/second - unlinked 0 (time 1761311480 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 832 (time 1761311561.27 total 10.00 last 83.18) - open/close 1579 (time 1761311571.27 total 20.00 last 74.69) - open/close 2373 (time 1761311581.28 total 30.01 last 79.34) - open/close 3171 (time 1761311591.28 total 40.01 last 79.79) - open/close 4006 (time 1761311601.29 total 50.02 last 83.43) - open/close 4821 (time 1761311611.30 total 60.03 last 81.43) - open/close 5618 (time 1761311621.30 total 70.04 last 79.64) - open/close 6435 (time 1761311631.31 total 80.04 last 81.68) - open/close 7281 (time 1761311641.31 total 90.04 last 84.56) - open/close 8094 (time 1761311651.32 total 100.05 last 81.22) - open/close 8876 (time 1761311661.33 total 110.06 last 78.17) - open/close 9670 (time 1761311671.33 total 120.06 last 79.38) total: 10000 open/close in 124.58 seconds: 80.27 ops/second - unlinked 0 (time 1761311679 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 759 (time 1761311773.33 total 10.00 last 75.88) - open/close 1529 (time 1761311783.34 total 20.01 last 76.98) - open/close 2316 (time 1761311793.34 total 30.01 last 78.66) - open/close 3074 (time 1761311803.35 total 40.01 last 75.77) - open/close 3850 (time 1761311813.35 total 50.02 last 77.55) - open/close 4638 (time 1761311823.36 total 60.03 last 78.72) - open/close 5425 (time 1761311833.36 total 70.03 last 78.68) - open/close 6160 (time 1761311843.37 total 80.04 last 73.45) - open/close 6929 (time 1761311853.37 total 90.04 last 76.88) - open/close 7628 (time 1761311863.38 total 100.05 last 69.88) - open/close 8387 (time 1761311873.39 total 110.06 last 75.81) - open/close 9147 (time 1761311883.40 total 120.07 last 75.92) - open/close 9886 (time 1761311893.41 total 130.08 last 73.85) total: 10000 open/close in 131.48 seconds: 76.06 ops/second - unlinked 0 (time 1761311897 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 761 (time 1761311985.33 total 10.00 last 76.10) - open/close 1571 (time 1761311995.33 total 20.00 last 81.00) - open/close 2358 (time 1761312005.34 total 30.01 last 78.65) - open/close 3156 (time 1761312015.35 total 40.02 last 79.70) - open/close 3916 (time 1761312025.35 total 50.02 last 75.99) - open/close 4718 (time 1761312035.36 total 60.03 last 80.15) - open/close 5484 (time 1761312045.36 total 70.03 last 76.58) - open/close 6240 (time 1761312055.37 total 80.04 last 75.55) - open/close 7036 (time 1761312065.38 total 90.04 last 79.53) - open/close 7819 (time 1761312075.38 total 100.05 last 78.24) - open/close 8612 (time 1761312085.39 total 110.06 last 79.26) - open/close 9444 (time 1761312095.40 total 120.06 last 83.14) total: 10000 open/close in 126.92 seconds: 78.79 ops/second - unlinked 0 (time 1761312106 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 789 (time 1761312193.00 total 10.01 last 78.81) - open/close 1658 (time 1761312203.01 total 20.01 last 86.88) - open/close 2464 (time 1761312213.01 total 30.02 last 80.55) - open/close 3296 (time 1761312223.01 total 40.02 last 83.17) - open/close 4103 (time 1761312233.01 total 50.02 last 80.70) - open/close 4879 (time 1761312243.01 total 60.02 last 77.60) - open/close 5631 (time 1761312253.02 total 70.03 last 75.17) - open/close 6437 (time 1761312263.02 total 80.03 last 80.58) - open/close 7235 (time 1761312273.03 total 90.04 last 79.74) - open/close 8017 (time 1761312283.03 total 100.04 last 78.18) - open/close 8805 (time 1761312293.04 total 110.05 last 78.75) - open/close 9516 (time 1761312303.07 total 120.08 last 70.87) total: 10000 open/close in 127.64 seconds: 78.35 ops/second - unlinked 0 (time 1761312315 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 827 (time 1761312393.85 total 10.00 last 82.69) - open/close 1689 (time 1761312403.86 total 20.01 last 86.11) - open/close 2566 (time 1761312413.87 total 30.02 last 87.65) - open/close 3431 (time 1761312423.88 total 40.03 last 86.42) - open/close 4205 (time 1761312433.88 total 50.03 last 77.36) - open/close 5004 (time 1761312443.89 total 60.04 last 79.84) - open/close 5863 (time 1761312453.89 total 70.04 last 85.88) - open/close 6746 (time 1761312463.90 total 80.05 last 88.21) - open/close 7604 (time 1761312473.91 total 90.06 last 85.76) - open/close 8459 (time 1761312483.92 total 100.07 last 85.43) - open/close 9305 (time 1761312493.92 total 110.07 last 84.60) total: 10000 open/close in 118.32 seconds: 84.52 ops/second - unlinked 0 (time 1761312505 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4172 82420 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236882 s, 4.4 MB/s fail_loc=0x80000204 Failing ost1 on oleg140-server Stopping /mnt/lustre-ost1 (opts:) on oleg140-server reboot facets: ost1 Failover ost1 to oleg140-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 1.82 seconds: 54.94 ops/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg140-server: LNET ready to unload oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.