== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:43:52 (1761313432) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 725 (time 1761313571.97 total 10.01 last 72.46) - open/close 1488 (time 1761313581.98 total 20.01 last 76.25) - open/close 2203 (time 1761313591.98 total 30.02 last 71.47) - open/close 2888 (time 1761313601.99 total 40.03 last 68.41) - open/close 3638 (time 1761313612.01 total 50.05 last 74.85) - open/close 4341 (time 1761313622.02 total 60.05 last 70.25) - open/close 5103 (time 1761313632.03 total 70.06 last 76.13) - open/close 5824 (time 1761313642.03 total 80.07 last 72.07) - open/close 6413 (time 1761313652.04 total 90.07 last 58.86) - open/close 7012 (time 1761313662.04 total 100.08 last 59.89) - open/close 7661 (time 1761313672.04 total 110.08 last 64.90) - open/close 8379 (time 1761313682.06 total 120.10 last 71.66) - open/close 9122 (time 1761313692.06 total 130.10 last 74.30) - open/close 9910 (time 1761313702.06 total 140.10 last 78.78) total: 10000 open/close in 141.04 seconds: 70.90 ops/second - unlinked 0 (time 1761313705 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 640 (time 1761313791.52 total 10.02 last 63.85) - open/close 1305 (time 1761313801.52 total 20.03 last 66.48) - open/close 1992 (time 1761313811.53 total 30.03 last 68.66) - open/close 2702 (time 1761313821.53 total 40.04 last 70.97) - open/close 3425 (time 1761313831.53 total 50.04 last 72.29) - open/close 4101 (time 1761313841.55 total 60.06 last 67.48) - open/close 4733 (time 1761313851.57 total 70.07 last 63.09) - open/close 5457 (time 1761313861.58 total 80.09 last 72.30) - open/close 6027 (time 1761313871.59 total 90.10 last 56.92) - open/close 6617 (time 1761313881.61 total 100.12 last 58.90) - open/close 7236 (time 1761313891.62 total 110.12 last 61.87) - open/close 7951 (time 1761313901.62 total 120.13 last 71.46) - open/close 8682 (time 1761313911.62 total 130.13 last 73.09) - open/close 9400 (time 1761313921.62 total 140.13 last 71.79) total: 10000 open/close in 148.40 seconds: 67.39 ops/second - unlinked 0 (time 1761313933 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 683 (time 1761314027.32 total 10.01 last 68.27) - open/close 1427 (time 1761314037.32 total 20.01 last 74.37) - open/close 2185 (time 1761314047.33 total 30.01 last 75.78) - open/close 2928 (time 1761314057.34 total 40.02 last 74.22) - open/close 3697 (time 1761314067.34 total 50.03 last 76.86) - open/close 4407 (time 1761314077.34 total 60.03 last 71.00) - open/close 5075 (time 1761314087.35 total 70.03 last 66.76) - open/close 5710 (time 1761314097.37 total 80.05 last 63.36) - open/close 6385 (time 1761314107.38 total 90.06 last 67.44) - open/close 7113 (time 1761314117.38 total 100.07 last 72.78) - open/close 7804 (time 1761314127.40 total 110.08 last 68.98) - open/close 8516 (time 1761314137.41 total 120.10 last 71.11) - open/close 9169 (time 1761314147.42 total 130.11 last 65.22) - open/close 9812 (time 1761314157.43 total 140.11 last 64.29) total: 10000 open/close in 142.72 seconds: 70.07 ops/second - unlinked 0 (time 1761314164 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 556 (time 1761314262.44 total 10.01 last 55.55) - open/close 1083 (time 1761314272.46 total 20.03 last 52.60) - open/close 1680 (time 1761314282.46 total 30.03 last 59.68) - open/close 2337 (time 1761314292.48 total 40.05 last 65.60) - open/close 3039 (time 1761314302.48 total 50.05 last 70.19) - open/close 3809 (time 1761314312.49 total 60.05 last 76.97) - open/close 4518 (time 1761314322.49 total 70.06 last 70.85) - open/close 5256 (time 1761314332.50 total 80.06 last 73.77) - open/close 6001 (time 1761314342.51 total 90.08 last 74.41) - open/close 6731 (time 1761314352.52 total 100.08 last 72.95) - open/close 7475 (time 1761314362.53 total 110.09 last 74.31) - open/close 8182 (time 1761314372.53 total 120.10 last 70.69) - open/close 8938 (time 1761314382.53 total 130.10 last 75.57) - open/close 9712 (time 1761314392.54 total 140.10 last 77.38) total: 10000 open/close in 143.93 seconds: 69.48 ops/second - unlinked 0 (time 1761314400 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 759 (time 1761314497.39 total 10.00 last 75.89) - open/close 1478 (time 1761314507.39 total 20.01 last 71.86) - open/close 2211 (time 1761314517.42 total 30.03 last 73.09) - open/close 2859 (time 1761314527.43 total 40.04 last 64.74) - open/close 3471 (time 1761314537.44 total 50.05 last 61.14) - open/close 4067 (time 1761314547.45 total 60.06 last 59.57) - open/close 4742 (time 1761314557.46 total 70.07 last 67.39) - open/close 5327 (time 1761314567.47 total 80.08 last 58.45) - open/close 5993 (time 1761314577.48 total 90.09 last 66.53) - open/close 6724 (time 1761314587.50 total 100.11 last 72.99) - open/close 7462 (time 1761314597.52 total 110.13 last 73.65) - open/close 8271 (time 1761314607.53 total 120.14 last 80.82) - open/close 9073 (time 1761314617.53 total 130.14 last 80.16) - open/close 9862 (time 1761314627.54 total 140.16 last 78.80) total: 10000 open/close in 142.12 seconds: 70.36 ops/second - unlinked 0 (time 1761314633 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 711 (time 1761314731.84 total 10.01 last 71.05) - open/close 1394 (time 1761314741.85 total 20.02 last 68.22) - open/close 2118 (time 1761314751.85 total 30.02 last 72.36) - open/close 2890 (time 1761314761.86 total 40.03 last 77.15) - open/close 3607 (time 1761314771.89 total 50.06 last 71.50) - open/close 4338 (time 1761314781.89 total 60.06 last 73.09) - open/close 5023 (time 1761314791.90 total 70.07 last 68.44) - open/close 5728 (time 1761314801.91 total 80.08 last 70.42) - open/close 6420 (time 1761314811.92 total 90.09 last 69.12) - open/close 7139 (time 1761314821.93 total 100.10 last 71.82) - open/close 7859 (time 1761314831.94 total 110.11 last 71.93) - open/close 8555 (time 1761314841.95 total 120.12 last 69.54) - open/close 9315 (time 1761314851.97 total 130.14 last 75.83) total: 10000 open/close in 139.28 seconds: 71.80 ops/second - unlinked 0 (time 1761314865 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 736 (time 1761314963.96 total 10.01 last 73.54) - open/close 1435 (time 1761314973.97 total 20.02 last 69.84) - open/close 2157 (time 1761314983.97 total 30.02 last 72.19) - open/close 2885 (time 1761314993.97 total 40.02 last 72.80) - open/close 3662 (time 1761315003.99 total 50.03 last 77.61) - open/close 4384 (time 1761315014.00 total 60.04 last 72.13) - open/close 5049 (time 1761315024.00 total 70.04 last 66.46) - open/close 5737 (time 1761315034.02 total 80.06 last 68.68) - open/close 6396 (time 1761315044.02 total 90.07 last 65.87) - open/close 7072 (time 1761315054.04 total 100.08 last 67.52) - open/close 7733 (time 1761315064.04 total 110.09 last 66.06) - open/close 8482 (time 1761315074.04 total 120.09 last 74.88) - open/close 9191 (time 1761315084.05 total 130.10 last 70.85) - open/close 9923 (time 1761315094.06 total 140.11 last 73.12) total: 10000 open/close in 141.14 seconds: 70.85 ops/second - unlinked 0 (time 1761315099 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 673 (time 1761315197.12 total 10.02 last 67.18) - open/close 1353 (time 1761315207.12 total 20.02 last 67.99) - open/close 1978 (time 1761315217.13 total 30.02 last 62.47) - open/close 2650 (time 1761315227.13 total 40.03 last 67.16) - open/close 3250 (time 1761315237.14 total 50.04 last 59.94) - open/close 3880 (time 1761315247.15 total 60.04 last 62.98) - open/close 4585 (time 1761315257.15 total 70.05 last 70.47) - open/close 5266 (time 1761315267.16 total 80.05 last 68.07) - open/close 5898 (time 1761315277.17 total 90.06 last 63.13) - open/close 6406 (time 1761315287.17 total 100.07 last 50.78) - open/close 6981 (time 1761315297.18 total 110.07 last 57.46) - open/close 7635 (time 1761315307.18 total 120.08 last 65.38) - open/close 8283 (time 1761315317.18 total 130.08 last 64.78) - open/close 8930 (time 1761315327.19 total 140.08 last 64.68) - open/close 9593 (time 1761315337.19 total 150.09 last 66.26) total: 10000 open/close in 157.02 seconds: 63.69 ops/second - unlinked 0 (time 1761315348 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 721 (time 1761315445.16 total 10.01 last 72.05) - open/close 1440 (time 1761315455.16 total 20.01 last 71.89) - open/close 2124 (time 1761315465.16 total 30.01 last 68.39) - open/close 2751 (time 1761315475.17 total 40.02 last 62.61) - open/close 3400 (time 1761315485.19 total 50.04 last 64.81) - open/close 4152 (time 1761315495.20 total 60.05 last 75.12) - open/close 4886 (time 1761315505.21 total 70.06 last 73.34) - open/close 5586 (time 1761315515.21 total 80.06 last 70.00) - open/close 6195 (time 1761315525.21 total 90.06 last 60.86) - open/close 6872 (time 1761315535.22 total 100.06 last 67.68) - open/close 7570 (time 1761315545.22 total 110.07 last 69.80) - open/close 8275 (time 1761315555.23 total 120.08 last 70.40) - open/close 8992 (time 1761315565.24 total 130.09 last 71.62) - open/close 9695 (time 1761315575.25 total 140.09 last 70.27) total: 10000 open/close in 144.60 seconds: 69.16 ops/second - unlinked 0 (time 1761315583 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 766 (time 1761315677.77 total 10.00 last 76.57) - open/close 1536 (time 1761315687.78 total 20.02 last 76.88) - open/close 2312 (time 1761315697.79 total 30.03 last 77.52) - open/close 3041 (time 1761315707.80 total 40.04 last 72.85) - open/close 3767 (time 1761315717.81 total 50.05 last 72.53) - open/close 4527 (time 1761315727.81 total 60.05 last 75.99) - open/close 5262 (time 1761315737.81 total 70.05 last 73.47) - open/close 5998 (time 1761315747.82 total 80.06 last 73.55) - open/close 6740 (time 1761315757.82 total 90.06 last 74.16) - open/close 7442 (time 1761315767.85 total 100.09 last 70.04) - open/close 8137 (time 1761315777.85 total 110.09 last 69.49) - open/close 8791 (time 1761315787.86 total 120.09 last 65.36) - open/close 9464 (time 1761315797.87 total 130.11 last 67.20) total: 10000 open/close in 138.20 seconds: 72.36 ops/second - unlinked 0 (time 1761315810 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 685 (time 1761315905.28 total 10.00 last 68.47) - open/close 1351 (time 1761315915.29 total 20.02 last 66.52) - open/close 2055 (time 1761315925.30 total 30.02 last 70.35) - open/close 2735 (time 1761315935.31 total 40.03 last 67.94) - open/close 3470 (time 1761315945.31 total 50.04 last 73.46) - open/close 4200 (time 1761315955.32 total 60.04 last 72.96) - open/close 4923 (time 1761315965.32 total 70.04 last 72.30) - open/close 5671 (time 1761315975.32 total 80.05 last 74.79) - open/close 6380 (time 1761315985.33 total 90.06 last 70.82) - open/close 7103 (time 1761315995.34 total 100.07 last 72.24) - open/close 7840 (time 1761316005.35 total 110.08 last 73.62) - open/close 8540 (time 1761316015.36 total 120.08 last 69.95) - open/close 9293 (time 1761316025.36 total 130.08 last 75.29) total: 10000 open/close in 139.77 seconds: 71.55 ops/second - unlinked 0 (time 1761316038 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4140 82452 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281174 s, 3.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server reboot facets: ost1 Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761316207.91 total 27.71 last 0.00) total: 100 open/close in 30.43 seconds: 3.29 ops/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.