== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 17:02:58 (1776200578) Stopping clients: oleg458-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:-f) oleg458-server: oleg458-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg458-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg458-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock oleg458-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 575 (time 1776200720.70 total 10.01 last 57.45) - open/close 1162 (time 1776200730.71 total 20.01 last 58.68) - open/close 1771 (time 1776200740.72 total 30.02 last 60.83) - open/close 2361 (time 1776200750.72 total 40.02 last 58.98) - open/close 2952 (time 1776200760.73 total 50.03 last 59.05) - open/close 3416 (time 1776200770.74 total 60.05 last 46.34) - open/close 3867 (time 1776200780.75 total 70.05 last 45.07) - open/close 4474 (time 1776200790.76 total 80.06 last 60.66) - open/close 4984 (time 1776200800.76 total 90.06 last 50.99) - open/close 5526 (time 1776200810.77 total 100.07 last 54.15) - open/close 6060 (time 1776200820.77 total 110.08 last 53.37) - open/close 6628 (time 1776200830.80 total 120.10 last 56.66) - open/close 7161 (time 1776200840.81 total 130.11 last 53.24) - open/close 7723 (time 1776200850.82 total 140.12 last 56.14) - open/close 8268 (time 1776200860.82 total 150.13 last 54.47) - open/close 8851 (time 1776200870.83 total 160.14 last 58.25) - open/close 9375 (time 1776200880.83 total 170.14 last 52.39) - open/close 9940 (time 1776200890.84 total 180.14 last 56.47) total: 10000 open/close in 181.14 seconds: 55.21 ops/second - unlinked 0 (time 1776200897 ; total 1 ; last 1) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 639 (time 1776201012.05 total 10.00 last 63.90) - open/close 1215 (time 1776201022.05 total 20.00 last 57.58) - open/close 1844 (time 1776201032.08 total 30.03 last 62.73) - open/close 2460 (time 1776201042.08 total 40.04 last 61.57) - open/close 3035 (time 1776201052.09 total 50.05 last 57.43) - open/close 3589 (time 1776201062.11 total 60.06 last 55.32) - open/close 4250 (time 1776201072.12 total 70.07 last 66.02) - open/close 4874 (time 1776201082.14 total 80.10 last 62.24) - open/close 5481 (time 1776201092.17 total 90.12 last 60.54) - open/close 6075 (time 1776201102.18 total 100.14 last 59.31) - open/close 6651 (time 1776201112.18 total 110.14 last 57.60) - open/close 7174 (time 1776201122.19 total 120.15 last 52.25) - open/close 7666 (time 1776201132.20 total 130.15 last 49.18) - open/close 8214 (time 1776201142.21 total 140.16 last 54.74) - open/close 8875 (time 1776201152.21 total 150.17 last 66.06) - open/close 9448 (time 1776201162.22 total 160.17 last 57.30) total: 10000 open/close in 169.90 seconds: 58.86 ops/second - unlinked 0 (time 1776201177 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 564 (time 1776201291.14 total 10.01 last 56.35) - open/close 1184 (time 1776201301.15 total 20.02 last 61.95) - open/close 1747 (time 1776201311.16 total 30.03 last 56.22) - open/close 2334 (time 1776201321.17 total 40.03 last 58.68) - open/close 2981 (time 1776201331.18 total 50.05 last 64.60) - open/close 3554 (time 1776201341.20 total 60.07 last 57.20) - open/close 4197 (time 1776201351.21 total 70.08 last 64.22) - open/close 4821 (time 1776201361.22 total 80.09 last 62.36) - open/close 5451 (time 1776201371.24 total 90.10 last 62.88) - open/close 6024 (time 1776201381.24 total 100.11 last 57.29) - open/close 6671 (time 1776201391.24 total 110.11 last 64.69) - open/close 7300 (time 1776201401.25 total 120.12 last 62.82) - open/close 7898 (time 1776201411.25 total 130.12 last 59.78) - open/close 8480 (time 1776201421.26 total 140.12 last 58.19) - open/close 9046 (time 1776201431.27 total 150.14 last 56.53) - open/close 9613 (time 1776201441.28 total 160.14 last 56.66) total: 10000 open/close in 166.82 seconds: 59.94 ops/second - unlinked 0 (time 1776201452 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 619 (time 1776201567.12 total 10.00 last 61.88) - open/close 1222 (time 1776201577.12 total 20.01 last 60.26) - open/close 1820 (time 1776201587.13 total 30.02 last 59.72) - open/close 2442 (time 1776201597.15 total 40.03 last 62.13) - open/close 3045 (time 1776201607.15 total 50.04 last 60.25) - open/close 3653 (time 1776201617.16 total 60.04 last 60.79) - open/close 4202 (time 1776201627.16 total 70.05 last 54.88) - open/close 4757 (time 1776201637.16 total 80.05 last 55.48) - open/close 5293 (time 1776201647.18 total 90.06 last 53.53) - open/close 5878 (time 1776201657.20 total 100.09 last 58.36) - open/close 6431 (time 1776201667.20 total 110.09 last 55.30) - open/close 6985 (time 1776201677.21 total 120.10 last 55.33) - open/close 7521 (time 1776201687.22 total 130.11 last 53.57) - open/close 8117 (time 1776201697.24 total 140.13 last 59.47) - open/close 8673 (time 1776201707.25 total 150.13 last 55.56) - open/close 9325 (time 1776201717.27 total 160.16 last 65.06) - open/close 9848 (time 1776201727.28 total 170.17 last 52.23) total: 10000 open/close in 173.27 seconds: 57.71 ops/second - unlinked 0 (time 1776201735 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 555 (time 1776201857.36 total 10.01 last 55.42) - open/close 1137 (time 1776201867.37 total 20.02 last 58.14) - open/close 1698 (time 1776201877.37 total 30.03 last 56.09) - open/close 2292 (time 1776201887.38 total 40.03 last 59.39) - open/close 2872 (time 1776201897.39 total 50.04 last 57.93) - open/close 3457 (time 1776201907.40 total 60.05 last 58.44) - open/close 4039 (time 1776201917.41 total 70.07 last 58.10) - open/close 4613 (time 1776201927.42 total 80.07 last 57.35) - open/close 5204 (time 1776201937.44 total 90.09 last 59.02) - open/close 5776 (time 1776201947.46 total 100.11 last 57.07) - open/close 6356 (time 1776201957.47 total 110.12 last 57.96) - open/close 6921 (time 1776201967.48 total 120.13 last 56.44) - open/close 7494 (time 1776201977.49 total 130.14 last 57.25) - open/close 8126 (time 1776201987.50 total 140.15 last 63.12) - open/close 8759 (time 1776201997.51 total 150.16 last 63.25) - open/close 9337 (time 1776202007.51 total 160.16 last 57.80) - open/close 9915 (time 1776202017.52 total 170.17 last 57.74) total: 10000 open/close in 171.64 seconds: 58.26 ops/second - unlinked 0 (time 1776202023 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 586 (time 1776202136.77 total 10.01 last 58.54) - open/close 1190 (time 1776202146.78 total 20.02 last 60.37) - open/close 1856 (time 1776202156.79 total 30.03 last 66.51) - open/close 2416 (time 1776202166.81 total 40.04 last 55.92) - open/close 2970 (time 1776202176.81 total 50.05 last 55.36) - open/close 3552 (time 1776202186.81 total 60.05 last 58.19) - open/close 4230 (time 1776202196.82 total 70.06 last 67.74) - open/close 4810 (time 1776202206.83 total 80.07 last 57.94) - open/close 5406 (time 1776202216.84 total 90.08 last 59.55) - open/close 6051 (time 1776202226.85 total 100.09 last 64.43) - open/close 6638 (time 1776202236.86 total 110.09 last 58.68) - open/close 7148 (time 1776202246.87 total 120.10 last 50.95) - open/close 7639 (time 1776202256.87 total 130.11 last 49.08) - open/close 8231 (time 1776202266.88 total 140.11 last 59.16) - open/close 8848 (time 1776202276.88 total 150.12 last 61.66) - open/close 9416 (time 1776202286.89 total 160.13 last 56.78) total: 10000 open/close in 169.13 seconds: 59.12 ops/second - unlinked 0 (time 1776202299 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 631 (time 1776202411.52 total 10.02 last 62.97) - open/close 1235 (time 1776202421.55 total 20.05 last 60.23) - open/close 1773 (time 1776202431.55 total 30.05 last 53.79) - open/close 2246 (time 1776202441.58 total 40.08 last 47.14) - open/close 2804 (time 1776202451.59 total 50.09 last 55.78) - open/close 3430 (time 1776202461.60 total 60.10 last 62.54) - open/close 4028 (time 1776202471.60 total 70.10 last 59.77) - open/close 4660 (time 1776202481.60 total 80.11 last 63.19) - open/close 5223 (time 1776202491.61 total 90.11 last 56.25) - open/close 5820 (time 1776202501.62 total 100.12 last 59.65) - open/close 6431 (time 1776202511.63 total 110.13 last 61.06) - open/close 6998 (time 1776202521.63 total 120.13 last 56.68) - open/close 7546 (time 1776202531.65 total 130.15 last 54.71) - open/close 8130 (time 1776202541.66 total 140.16 last 58.35) - open/close 8694 (time 1776202551.67 total 150.17 last 56.32) - open/close 9255 (time 1776202561.68 total 160.18 last 56.04) - open/close 9778 (time 1776202571.69 total 170.20 last 52.23) total: 10000 open/close in 173.46 seconds: 57.65 ops/second - unlinked 0 (time 1776202579 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 621 (time 1776202688.21 total 10.01 last 62.06) - open/close 1316 (time 1776202698.23 total 20.02 last 69.37) - open/close 1972 (time 1776202708.23 total 30.03 last 65.57) - open/close 2565 (time 1776202718.24 total 40.04 last 59.25) - open/close 3192 (time 1776202728.25 total 50.05 last 62.65) - open/close 3803 (time 1776202738.26 total 60.06 last 61.03) - open/close 4428 (time 1776202748.28 total 70.08 last 62.37) - open/close 5026 (time 1776202758.29 total 80.09 last 59.73) - open/close 5620 (time 1776202768.29 total 90.09 last 59.37) - open/close 6206 (time 1776202778.30 total 100.10 last 58.58) - open/close 6825 (time 1776202788.30 total 110.10 last 61.88) - open/close 7371 (time 1776202798.30 total 120.10 last 54.60) - open/close 7968 (time 1776202808.32 total 130.12 last 59.59) - open/close 8518 (time 1776202818.33 total 140.13 last 54.92) - open/close 9142 (time 1776202828.34 total 150.13 last 62.38) - open/close 9727 (time 1776202838.34 total 160.14 last 58.49) total: 10000 open/close in 164.46 seconds: 60.81 ops/second - unlinked 0 (time 1776202847 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 581 (time 1776202957.44 total 10.00 last 58.09) - open/close 1165 (time 1776202967.46 total 20.02 last 58.30) - open/close 1797 (time 1776202977.46 total 30.02 last 63.18) - open/close 2435 (time 1776202987.47 total 40.03 last 63.76) - open/close 3070 (time 1776202997.47 total 50.03 last 63.47) - open/close 3682 (time 1776203007.49 total 60.05 last 61.09) - open/close 4292 (time 1776203017.54 total 70.10 last 60.69) - open/close 4930 (time 1776203027.55 total 80.11 last 63.76) - open/close 5506 (time 1776203037.56 total 90.13 last 57.51) - open/close 6085 (time 1776203047.57 total 100.13 last 57.86) - open/close 6660 (time 1776203057.58 total 110.14 last 57.44) - open/close 7227 (time 1776203067.59 total 120.16 last 56.62) - open/close 7838 (time 1776203077.60 total 130.17 last 61.04) - open/close 8420 (time 1776203087.60 total 140.17 last 58.20) - open/close 8998 (time 1776203097.61 total 150.17 last 57.75) - open/close 9553 (time 1776203107.61 total 160.18 last 55.48) total: 10000 open/close in 169.39 seconds: 59.04 ops/second - unlinked 0 (time 1776203121 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 567 (time 1776203240.18 total 10.01 last 56.63) - open/close 1186 (time 1776203250.20 total 20.03 last 61.80) - open/close 1809 (time 1776203260.21 total 30.04 last 62.22) - open/close 2399 (time 1776203270.22 total 40.05 last 58.96) - open/close 2890 (time 1776203280.25 total 50.08 last 48.92) - open/close 3445 (time 1776203290.27 total 60.10 last 55.41) - open/close 4060 (time 1776203300.27 total 70.11 last 61.47) - open/close 4625 (time 1776203310.28 total 80.12 last 56.44) - open/close 5249 (time 1776203320.30 total 90.13 last 62.29) - open/close 5791 (time 1776203330.32 total 100.15 last 54.09) - open/close 6358 (time 1776203340.33 total 110.16 last 56.67) - open/close 6958 (time 1776203350.36 total 120.19 last 59.82) - open/close 7453 (time 1776203360.37 total 130.20 last 49.44) - open/close 8026 (time 1776203370.38 total 140.21 last 57.26) - open/close 8547 (time 1776203380.39 total 150.23 last 52.02) - open/close 9120 (time 1776203390.40 total 160.23 last 57.27) - open/close 9569 (time 1776203400.40 total 170.23 last 44.88) total: 10000 open/close in 178.47 seconds: 56.03 ops/second - unlinked 0 (time 1776203413 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 492 (time 1776203543.03 total 10.01 last 49.17) - open/close 1072 (time 1776203553.04 total 20.01 last 57.95) - open/close 1634 (time 1776203563.05 total 30.02 last 56.17) - open/close 2232 (time 1776203573.05 total 40.03 last 59.75) - open/close 2811 (time 1776203583.07 total 50.04 last 57.83) - open/close 3428 (time 1776203593.07 total 60.04 last 61.68) - open/close 4012 (time 1776203603.08 total 70.06 last 58.32) - open/close 4532 (time 1776203613.11 total 80.08 last 51.87) - open/close 5093 (time 1776203623.11 total 90.08 last 56.09) - open/close 5693 (time 1776203633.12 total 100.10 last 59.92) - open/close 6286 (time 1776203643.14 total 110.11 last 59.24) - open/close 6862 (time 1776203653.14 total 120.11 last 57.58) - open/close 7448 (time 1776203663.15 total 130.12 last 58.55) - open/close 7978 (time 1776203673.16 total 140.13 last 52.96) - open/close 8522 (time 1776203683.19 total 150.16 last 54.23) - open/close 9062 (time 1776203693.21 total 160.18 last 53.89) - open/close 9601 (time 1776203703.23 total 170.20 last 53.78) total: 10000 open/close in 177.82 seconds: 56.24 ops/second - unlinked 0 (time 1776203716 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409475 s, 2.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg458-server Stopping /mnt/lustre-ost1 (opts:) on oleg458-server reboot facets: ost1 Failover ost1 to oleg458-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776203916.22 total 35.06 last 0.00) total: 100 open/close in 37.40 seconds: 2.67 ops/second umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg458-server: LNET ready to unload oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.