== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 11:15:21 (1776266121) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg411-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg411-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock oleg411-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 490 (time 1776266272.60 total 10.01 last 48.94) - open/close 946 (time 1776266282.62 total 20.03 last 45.50) - open/close 1548 (time 1776266292.63 total 30.05 last 60.11) - open/close 2236 (time 1776266302.64 total 40.06 last 68.75) - open/close 2932 (time 1776266312.66 total 50.07 last 69.47) - open/close 3673 (time 1776266322.67 total 60.09 last 73.99) - open/close 4382 (time 1776266332.67 total 70.09 last 70.89) - open/close 4966 (time 1776266342.69 total 80.11 last 58.29) - open/close 5587 (time 1776266352.71 total 90.13 last 62.00) - open/close 6223 (time 1776266362.72 total 100.14 last 63.53) - open/close 6815 (time 1776266372.72 total 110.14 last 59.19) - open/close 7401 (time 1776266382.75 total 120.16 last 58.45) - open/close 8022 (time 1776266392.75 total 130.17 last 62.05) - open/close 8617 (time 1776266402.78 total 140.20 last 59.35) - open/close 9264 (time 1776266412.80 total 150.22 last 64.57) - open/close 9865 (time 1776266422.84 total 160.25 last 59.89) total: 10000 open/close in 162.24 seconds: 61.64 ops/second - unlinked 0 (time 1776266428 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 583 (time 1776266540.68 total 10.00 last 58.30) - open/close 1117 (time 1776266550.68 total 20.01 last 53.37) - open/close 1716 (time 1776266560.71 total 30.03 last 59.75) - open/close 2345 (time 1776266570.72 total 40.04 last 62.85) - open/close 2881 (time 1776266580.74 total 50.07 last 53.46) - open/close 3604 (time 1776266590.75 total 60.07 last 72.27) - open/close 4269 (time 1776266600.75 total 70.08 last 66.46) - open/close 4898 (time 1776266610.77 total 80.09 last 62.79) - open/close 5574 (time 1776266620.79 total 90.11 last 67.47) - open/close 6236 (time 1776266630.81 total 100.13 last 66.08) - open/close 6864 (time 1776266640.81 total 110.13 last 62.78) - open/close 7587 (time 1776266650.82 total 120.14 last 72.23) - open/close 8248 (time 1776266660.83 total 130.15 last 66.06) - open/close 8876 (time 1776266670.84 total 140.16 last 62.72) - open/close 9487 (time 1776266680.84 total 150.17 last 61.08) total: 10000 open/close in 157.74 seconds: 63.39 ops/second - unlinked 0 (time 1776266692 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 562 (time 1776266807.56 total 10.02 last 56.10) - open/close 1265 (time 1776266817.56 total 20.02 last 70.29) - open/close 1785 (time 1776266827.58 total 30.04 last 51.87) - open/close 2377 (time 1776266837.59 total 40.05 last 59.16) - open/close 3021 (time 1776266847.60 total 50.06 last 64.34) - open/close 3634 (time 1776266857.61 total 60.07 last 61.21) - open/close 4263 (time 1776266867.61 total 70.08 last 62.89) - open/close 4910 (time 1776266877.62 total 80.09 last 64.63) - open/close 5548 (time 1776266887.63 total 90.09 last 63.76) - open/close 6215 (time 1776266897.63 total 100.10 last 66.68) - open/close 6900 (time 1776266907.65 total 110.11 last 68.41) - open/close 7672 (time 1776266917.65 total 120.11 last 77.19) - open/close 8409 (time 1776266927.66 total 130.12 last 73.62) - open/close 8979 (time 1776266937.67 total 140.13 last 56.95) - open/close 9570 (time 1776266947.67 total 150.13 last 59.10) total: 10000 open/close in 156.45 seconds: 63.92 ops/second - unlinked 0 (time 1776266958 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 606 (time 1776267065.98 total 10.01 last 60.56) - open/close 1159 (time 1776267075.99 total 20.02 last 55.25) - open/close 1663 (time 1776267086.01 total 30.03 last 50.31) - open/close 2154 (time 1776267096.02 total 40.04 last 49.05) - open/close 2783 (time 1776267106.03 total 50.06 last 62.83) - open/close 3416 (time 1776267116.04 total 60.06 last 63.26) - open/close 3970 (time 1776267126.05 total 70.07 last 55.34) - open/close 4561 (time 1776267136.05 total 80.08 last 59.05) - open/close 5192 (time 1776267146.10 total 90.13 last 62.79) - open/close 5811 (time 1776267156.11 total 100.14 last 61.86) - open/close 6424 (time 1776267166.11 total 110.14 last 61.29) - open/close 6871 (time 1776267176.12 total 120.15 last 44.65) - open/close 7459 (time 1776267186.13 total 130.15 last 58.78) - open/close 8056 (time 1776267196.14 total 140.17 last 59.61) - open/close 8613 (time 1776267206.16 total 150.18 last 55.61) - open/close 9217 (time 1776267216.20 total 160.23 last 60.12) - open/close 9764 (time 1776267226.21 total 170.24 last 54.64) total: 10000 open/close in 174.57 seconds: 57.28 ops/second - unlinked 0 (time 1776267236 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 632 (time 1776267348.54 total 10.01 last 63.15) - open/close 1277 (time 1776267358.54 total 20.01 last 64.47) - open/close 1923 (time 1776267368.55 total 30.02 last 64.57) - open/close 2581 (time 1776267378.56 total 40.03 last 65.71) - open/close 3209 (time 1776267388.57 total 50.04 last 62.74) - open/close 3779 (time 1776267398.61 total 60.08 last 56.75) - open/close 4340 (time 1776267408.61 total 70.08 last 56.10) - open/close 4845 (time 1776267418.62 total 80.09 last 50.48) - open/close 5496 (time 1776267428.64 total 90.11 last 64.97) - open/close 6229 (time 1776267438.64 total 100.11 last 73.30) - open/close 6742 (time 1776267448.65 total 110.12 last 51.22) - open/close 7410 (time 1776267458.66 total 120.13 last 66.77) - open/close 7985 (time 1776267468.72 total 130.18 last 57.17) - open/close 8467 (time 1776267478.73 total 140.20 last 48.13) - open/close 8917 (time 1776267488.75 total 150.22 last 44.92) - open/close 9488 (time 1776267498.76 total 160.23 last 57.05) - open/close 9998 (time 1776267508.76 total 170.23 last 50.99) total: 10000 open/close in 170.24 seconds: 58.74 ops/second - unlinked 0 (time 1776267514 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 752 (time 1776267628.24 total 10.00 last 75.19) - open/close 1374 (time 1776267638.25 total 20.01 last 62.16) - open/close 2006 (time 1776267648.25 total 30.01 last 63.20) - open/close 2645 (time 1776267658.26 total 40.02 last 63.84) - open/close 3246 (time 1776267668.28 total 50.04 last 59.98) - open/close 3933 (time 1776267678.28 total 60.04 last 68.69) - open/close 4686 (time 1776267688.28 total 70.05 last 75.25) - open/close 5194 (time 1776267698.30 total 80.06 last 50.74) - open/close 5817 (time 1776267708.32 total 90.08 last 62.16) - open/close 6507 (time 1776267718.32 total 100.09 last 68.96) - open/close 7134 (time 1776267728.33 total 110.09 last 62.69) - open/close 7942 (time 1776267738.33 total 120.09 last 80.79) - open/close 8635 (time 1776267748.34 total 130.10 last 69.25) - open/close 9311 (time 1776267758.34 total 140.10 last 67.58) total: 10000 open/close in 150.03 seconds: 66.65 ops/second - unlinked 0 (time 1776267773 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 486 (time 1776267894.65 total 10.01 last 48.57) - open/close 1013 (time 1776267904.67 total 20.03 last 52.55) - open/close 1527 (time 1776267914.68 total 30.04 last 51.36) - open/close 1923 (time 1776267924.70 total 40.06 last 39.53) - open/close 2472 (time 1776267934.70 total 50.06 last 54.89) - open/close 2868 (time 1776267944.71 total 60.07 last 39.56) - open/close 3257 (time 1776267954.73 total 70.09 last 38.85) - open/close 3726 (time 1776267964.73 total 80.09 last 46.87) - open/close 4239 (time 1776267974.74 total 90.10 last 51.27) - open/close 4823 (time 1776267984.74 total 100.10 last 58.36) - open/close 5414 (time 1776267994.76 total 110.12 last 59.00) - open/close 6002 (time 1776268004.77 total 120.13 last 58.77) - open/close 6622 (time 1776268014.77 total 130.13 last 61.98) - open/close 7115 (time 1776268024.78 total 140.14 last 49.25) - open/close 7676 (time 1776268034.80 total 150.16 last 56.01) - open/close 8278 (time 1776268044.81 total 160.18 last 60.09) - open/close 8876 (time 1776268054.82 total 170.18 last 59.79) - open/close 9514 (time 1776268064.83 total 180.19 last 63.75) total: 10000 open/close in 188.32 seconds: 53.10 ops/second - unlinked 0 (time 1776268077 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 533 (time 1776268202.01 total 10.01 last 53.24) - open/close 1135 (time 1776268212.01 total 20.01 last 60.19) - open/close 1647 (time 1776268222.01 total 30.01 last 51.19) - open/close 2242 (time 1776268232.03 total 40.03 last 59.39) - open/close 2834 (time 1776268242.04 total 50.04 last 59.17) - open/close 3370 (time 1776268252.04 total 60.04 last 53.56) - open/close 3947 (time 1776268262.05 total 70.05 last 57.66) - open/close 4471 (time 1776268272.07 total 80.07 last 52.31) - open/close 5117 (time 1776268282.08 total 90.08 last 64.55) - open/close 5773 (time 1776268292.09 total 100.09 last 65.54) - open/close 6390 (time 1776268302.09 total 110.09 last 61.67) - open/close 6989 (time 1776268312.10 total 120.10 last 59.87) - open/close 7565 (time 1776268322.11 total 130.11 last 57.53) - open/close 8096 (time 1776268332.12 total 140.12 last 53.05) - open/close 8686 (time 1776268342.12 total 150.12 last 58.98) - open/close 9211 (time 1776268352.14 total 160.14 last 52.40) - open/close 9803 (time 1776268362.15 total 170.15 last 59.11) total: 10000 open/close in 173.38 seconds: 57.68 ops/second - unlinked 0 (time 1776268370 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 642 (time 1776268487.59 total 10.01 last 64.13) - open/close 1280 (time 1776268497.60 total 20.02 last 63.73) - open/close 1851 (time 1776268507.60 total 30.02 last 57.08) - open/close 2410 (time 1776268517.62 total 40.05 last 55.76) - open/close 3014 (time 1776268527.64 total 50.07 last 60.29) - open/close 3555 (time 1776268537.67 total 60.10 last 53.95) - open/close 4173 (time 1776268547.67 total 70.10 last 61.79) - open/close 4750 (time 1776268557.67 total 80.10 last 57.69) - open/close 5386 (time 1776268567.69 total 90.11 last 63.53) - open/close 6016 (time 1776268577.70 total 100.13 last 62.90) - open/close 6559 (time 1776268587.71 total 110.14 last 54.24) - open/close 7121 (time 1776268597.71 total 120.14 last 56.19) - open/close 7606 (time 1776268607.73 total 130.15 last 48.44) - open/close 8060 (time 1776268617.73 total 140.16 last 45.37) - open/close 8520 (time 1776268627.74 total 150.17 last 45.96) - open/close 8998 (time 1776268637.75 total 160.17 last 47.76) - open/close 9632 (time 1776268647.77 total 170.19 last 63.30) total: 10000 open/close in 175.99 seconds: 56.82 ops/second - unlinked 0 (time 1776268659 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 584 (time 1776268782.62 total 10.01 last 58.37) - open/close 1163 (time 1776268792.63 total 20.01 last 57.88) - open/close 1755 (time 1776268802.64 total 30.02 last 59.11) - open/close 2333 (time 1776268812.68 total 40.06 last 57.60) - open/close 2774 (time 1776268822.68 total 50.07 last 44.07) - open/close 3339 (time 1776268832.69 total 60.08 last 56.44) - open/close 3990 (time 1776268842.70 total 70.08 last 65.05) - open/close 4588 (time 1776268852.71 total 80.09 last 59.74) - open/close 5098 (time 1776268862.72 total 90.11 last 50.94) - open/close 5723 (time 1776268872.73 total 100.11 last 62.47) - open/close 6465 (time 1776268882.73 total 110.11 last 74.17) - open/close 7078 (time 1776268892.74 total 120.12 last 61.28) - open/close 7715 (time 1776268902.74 total 130.12 last 63.69) - open/close 8487 (time 1776268912.74 total 140.12 last 77.19) - open/close 9176 (time 1776268922.74 total 150.12 last 68.87) - open/close 9921 (time 1776268932.76 total 160.14 last 74.37) total: 10000 open/close in 161.17 seconds: 62.05 ops/second - unlinked 0 (time 1776268938 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 434 (time 1776269049.71 total 10.02 last 43.32) - open/close 802 (time 1776269059.71 total 20.02 last 36.79) - open/close 1491 (time 1776269069.71 total 30.03 last 68.87) - open/close 2093 (time 1776269079.72 total 40.03 last 60.16) - open/close 2761 (time 1776269089.72 total 50.04 last 66.78) - open/close 3531 (time 1776269099.73 total 60.04 last 76.94) - open/close 4156 (time 1776269109.74 total 70.05 last 62.46) - open/close 4767 (time 1776269119.75 total 80.06 last 61.03) - open/close 5494 (time 1776269129.75 total 90.06 last 72.69) - open/close 6125 (time 1776269139.75 total 100.07 last 63.08) - open/close 6806 (time 1776269149.77 total 110.08 last 68.00) - open/close 7460 (time 1776269159.79 total 120.10 last 65.27) - open/close 8173 (time 1776269169.80 total 130.11 last 71.22) - open/close 8858 (time 1776269179.80 total 140.12 last 68.46) - open/close 9490 (time 1776269189.81 total 150.12 last 63.15) total: 10000 open/close in 159.35 seconds: 62.76 ops/second - unlinked 0 (time 1776269203 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.68295 s, 1.5 MB/s fail_loc=0x80000204 Failing ost1 on oleg411-server Stopping /mnt/lustre-ost1 (opts:) on oleg411-server reboot facets: ost1 Failover ost1 to oleg411-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776269388.57 total 35.48 last 0.00) total: 100 open/close in 38.38 seconds: 2.61 ops/second umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg411-server: LNET ready to unload oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.