== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 11:20:24 (1776266424) Stopping clients: oleg648-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg648-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg648-client: oleg648-server: ssh exited with exit code 2 oleg648-server: oleg648-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing load_modules_local oleg648-server: Loading modules from /home/green/git/lustre-release/lustre oleg648-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg648-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg648-server: detected 4 online CPUs by sysfs oleg648-server: Force libcfs to create 2 CPU partitions oleg648-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg648-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing load_modules_local oleg648-server: Loading modules from /home/green/git/lustre-release/lustre oleg648-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg648-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg648-server: detected 4 online CPUs by sysfs oleg648-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg648-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg648-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg648-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg648-client.virtnet: -o user_xattr,flock oleg648-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 794 (time 1776266592.78 total 10.00 last 79.38) - open/close 1552 (time 1776266602.78 total 20.01 last 75.77) - open/close 2420 (time 1776266612.80 total 30.02 last 86.69) - open/close 3236 (time 1776266622.80 total 40.02 last 81.59) - open/close 3959 (time 1776266632.80 total 50.02 last 72.30) - open/close 4761 (time 1776266642.80 total 60.02 last 80.17) - open/close 5736 (time 1776266652.80 total 70.02 last 97.50) - open/close 6536 (time 1776266662.81 total 80.03 last 79.95) - open/close 7311 (time 1776266672.81 total 90.03 last 77.47) - open/close 8138 (time 1776266682.81 total 100.03 last 82.70) - open/close 8947 (time 1776266692.83 total 110.05 last 80.78) - open/close 9796 (time 1776266702.83 total 120.05 last 84.88) total: 10000 open/close in 122.75 seconds: 81.46 ops/second - unlinked 0 (time 1776266709 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 851 (time 1776266797.79 total 10.00 last 85.09) - open/close 1737 (time 1776266807.80 total 20.01 last 88.49) - open/close 2622 (time 1776266817.80 total 30.02 last 88.48) - open/close 3441 (time 1776266827.83 total 40.04 last 81.71) - open/close 4251 (time 1776266837.84 total 50.05 last 80.90) - open/close 5072 (time 1776266847.85 total 60.06 last 81.99) - open/close 5910 (time 1776266857.86 total 70.07 last 83.76) - open/close 6699 (time 1776266867.87 total 80.08 last 78.82) - open/close 7512 (time 1776266877.87 total 90.08 last 81.26) - open/close 8292 (time 1776266887.88 total 100.09 last 77.94) - open/close 9075 (time 1776266897.89 total 110.11 last 78.19) - open/close 9843 (time 1776266907.90 total 120.12 last 76.72) total: 10000 open/close in 121.97 seconds: 81.98 ops/second - unlinked 0 (time 1776266913 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 810 (time 1776266999.24 total 10.01 last 80.90) - open/close 1697 (time 1776267009.24 total 20.01 last 88.70) - open/close 2528 (time 1776267019.25 total 30.02 last 83.03) - open/close 3355 (time 1776267029.26 total 40.04 last 82.55) - open/close 4127 (time 1776267039.27 total 50.04 last 77.18) - open/close 4929 (time 1776267049.28 total 60.05 last 80.11) - open/close 5771 (time 1776267059.28 total 70.06 last 84.16) - open/close 6528 (time 1776267069.28 total 80.06 last 75.70) - open/close 7319 (time 1776267079.29 total 90.06 last 79.06) - open/close 8137 (time 1776267089.29 total 100.07 last 81.78) - open/close 8938 (time 1776267099.29 total 110.07 last 80.08) - open/close 9655 (time 1776267109.31 total 120.09 last 71.55) total: 10000 open/close in 124.32 seconds: 80.44 ops/second - unlinked 0 (time 1776267117 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 822 (time 1776267201.73 total 10.02 last 82.04) - open/close 1652 (time 1776267211.74 total 20.03 last 82.93) - open/close 2332 (time 1776267221.76 total 30.05 last 67.87) - open/close 3048 (time 1776267231.77 total 40.05 last 71.55) - open/close 3879 (time 1776267241.78 total 50.06 last 83.03) - open/close 4651 (time 1776267251.80 total 60.08 last 77.06) - open/close 5399 (time 1776267261.83 total 70.12 last 74.52) - open/close 6128 (time 1776267271.84 total 80.13 last 72.83) - open/close 7016 (time 1776267281.84 total 90.13 last 88.80) - open/close 7893 (time 1776267291.85 total 100.13 last 87.67) - open/close 8744 (time 1776267301.85 total 110.14 last 85.03) - open/close 9597 (time 1776267311.86 total 120.14 last 85.28) total: 10000 open/close in 124.86 seconds: 80.09 ops/second - unlinked 0 (time 1776267320 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 524 (time 1776267409.26 total 10.00 last 52.40) - open/close 1114 (time 1776267419.26 total 20.00 last 59.00) - open/close 1706 (time 1776267429.28 total 30.01 last 59.13) - open/close 2335 (time 1776267439.28 total 40.02 last 62.88) - open/close 2959 (time 1776267449.28 total 50.02 last 62.38) - open/close 3710 (time 1776267459.29 total 60.03 last 75.06) - open/close 4448 (time 1776267469.31 total 70.05 last 73.61) - open/close 5239 (time 1776267479.34 total 80.08 last 78.88) - open/close 5987 (time 1776267489.35 total 90.09 last 74.76) - open/close 6701 (time 1776267499.36 total 100.09 last 71.35) - open/close 7537 (time 1776267509.37 total 110.10 last 83.50) - open/close 8367 (time 1776267519.37 total 120.11 last 82.95) - open/close 9203 (time 1776267529.37 total 130.11 last 83.60) - open/close 9962 (time 1776267539.38 total 140.12 last 75.84) total: 10000 open/close in 140.56 seconds: 71.14 ops/second - unlinked 0 (time 1776267544 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 765 (time 1776267635.29 total 10.02 last 76.32) - open/close 1562 (time 1776267645.32 total 20.05 last 79.46) - open/close 2402 (time 1776267655.33 total 30.06 last 83.93) - open/close 3188 (time 1776267665.35 total 40.08 last 78.47) - open/close 4029 (time 1776267675.35 total 50.08 last 84.08) - open/close 4758 (time 1776267685.37 total 60.10 last 72.79) - open/close 5490 (time 1776267695.37 total 70.10 last 73.15) - open/close 6276 (time 1776267705.38 total 80.11 last 78.52) - open/close 7044 (time 1776267715.39 total 90.12 last 76.77) - open/close 7864 (time 1776267725.39 total 100.12 last 81.98) - open/close 8658 (time 1776267735.39 total 110.12 last 79.35) - open/close 9398 (time 1776267745.40 total 120.13 last 73.94) total: 10000 open/close in 128.11 seconds: 78.06 ops/second - unlinked 0 (time 1776267757 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 776 (time 1776267851.75 total 10.00 last 77.57) - open/close 1575 (time 1776267861.77 total 20.02 last 79.81) - open/close 2367 (time 1776267871.77 total 30.02 last 79.18) - open/close 3185 (time 1776267881.77 total 40.02 last 81.76) - open/close 3956 (time 1776267891.78 total 50.03 last 77.04) - open/close 4742 (time 1776267901.78 total 60.03 last 78.58) - open/close 5561 (time 1776267911.79 total 70.04 last 81.86) - open/close 6306 (time 1776267921.79 total 80.04 last 74.49) - open/close 7127 (time 1776267931.79 total 90.04 last 82.08) - open/close 7882 (time 1776267941.80 total 100.05 last 75.46) - open/close 8600 (time 1776267951.80 total 110.05 last 71.75) - open/close 9302 (time 1776267961.81 total 120.06 last 70.19) total: 10000 open/close in 129.82 seconds: 77.03 ops/second - unlinked 0 (time 1776267976 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 776 (time 1776268070.17 total 10.02 last 77.48) - open/close 1579 (time 1776268080.18 total 20.02 last 80.26) - open/close 2287 (time 1776268090.18 total 30.02 last 70.77) - open/close 3002 (time 1776268100.20 total 40.04 last 71.36) - open/close 3749 (time 1776268110.20 total 50.05 last 74.67) - open/close 4524 (time 1776268120.23 total 60.07 last 77.29) - open/close 5259 (time 1776268130.23 total 70.08 last 73.46) - open/close 6068 (time 1776268140.24 total 80.08 last 80.87) - open/close 6817 (time 1776268150.25 total 90.09 last 74.84) - open/close 7619 (time 1776268160.26 total 100.11 last 80.09) - open/close 8420 (time 1776268170.26 total 110.11 last 80.07) - open/close 9148 (time 1776268180.27 total 120.12 last 72.73) - open/close 9913 (time 1776268190.28 total 130.12 last 76.47) total: 10000 open/close in 130.96 seconds: 76.36 ops/second - unlinked 0 (time 1776268195 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 806 (time 1776268286.00 total 10.00 last 80.57) - open/close 1532 (time 1776268296.01 total 20.02 last 72.51) - open/close 2296 (time 1776268306.02 total 30.03 last 76.28) - open/close 3008 (time 1776268316.03 total 40.03 last 71.18) - open/close 3873 (time 1776268326.03 total 50.04 last 86.46) - open/close 4492 (time 1776268336.04 total 60.05 last 61.85) - open/close 5303 (time 1776268346.04 total 70.05 last 81.10) - open/close 6127 (time 1776268356.05 total 80.06 last 82.28) - open/close 6904 (time 1776268366.06 total 90.07 last 77.64) - open/close 7679 (time 1776268376.07 total 100.08 last 77.44) - open/close 8496 (time 1776268386.07 total 110.08 last 81.66) - open/close 9330 (time 1776268396.08 total 120.08 last 83.37) total: 10000 open/close in 128.19 seconds: 78.01 ops/second - unlinked 0 (time 1776268408 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 783 (time 1776268501.77 total 10.02 last 78.15) - open/close 1575 (time 1776268511.78 total 20.03 last 79.09) - open/close 2390 (time 1776268521.80 total 30.05 last 81.33) - open/close 3176 (time 1776268531.80 total 40.05 last 78.59) - open/close 3978 (time 1776268541.80 total 50.05 last 80.20) - open/close 4735 (time 1776268551.81 total 60.07 last 75.62) - open/close 5473 (time 1776268561.82 total 70.07 last 73.80) - open/close 6301 (time 1776268571.82 total 80.08 last 82.73) - open/close 7067 (time 1776268581.83 total 90.08 last 76.58) - open/close 7813 (time 1776268591.83 total 100.08 last 74.59) - open/close 8617 (time 1776268601.83 total 110.08 last 80.39) - open/close 9324 (time 1776268611.83 total 120.09 last 70.67) - open/close 9984 (time 1776268621.83 total 130.09 last 66.00) total: 10000 open/close in 130.31 seconds: 76.74 ops/second - unlinked 0 (time 1776268626 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 815 (time 1776268720.99 total 10.01 last 81.44) - open/close 1634 (time 1776268731.01 total 20.03 last 81.71) - open/close 2418 (time 1776268741.02 total 30.04 last 78.31) - open/close 3173 (time 1776268751.02 total 40.04 last 75.48) - open/close 3883 (time 1776268761.03 total 50.05 last 70.94) - open/close 4498 (time 1776268771.04 total 60.06 last 61.45) - open/close 5107 (time 1776268781.05 total 70.07 last 60.83) - open/close 5790 (time 1776268791.06 total 80.08 last 68.25) - open/close 6485 (time 1776268801.07 total 90.09 last 69.44) - open/close 7147 (time 1776268811.07 total 100.09 last 66.19) - open/close 7784 (time 1776268821.08 total 110.10 last 63.65) - open/close 8477 (time 1776268831.09 total 120.11 last 69.21) - open/close 9297 (time 1776268841.10 total 130.12 last 81.90) total: 10000 open/close in 138.20 seconds: 72.36 ops/second - unlinked 0 (time 1776268853 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4184 82408 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1648 126568 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1648 126568 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211939 s, 4.9 MB/s fail_loc=0x80000204 Failing ost1 on oleg648-server Stopping /mnt/lustre-ost1 (opts:) on oleg648-server reboot facets: ost1 Failover ost1 to oleg648-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-OST0000 oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776269009.03 total 32.41 last 0.00) total: 100 open/close in 34.46 seconds: 2.90 ops/second umount lustre on /mnt/lustre..... Stopping client oleg648-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg648-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg648-server stop mds service on oleg648-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg648-server stop mds service on oleg648-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg648-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing unload_modules_local oleg648-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg648-server: LNET ready to unload oleg648-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.