== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 15:38:11 (1776195491) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg407-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock oleg407-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 619 (time 1776195621.53 total 10.01 last 61.85) - open/close 1298 (time 1776195631.53 total 20.01 last 67.85) - open/close 1979 (time 1776195641.55 total 30.03 last 68.01) - open/close 2672 (time 1776195651.56 total 40.04 last 69.19) - open/close 3301 (time 1776195661.57 total 50.05 last 62.84) - open/close 3952 (time 1776195671.58 total 60.06 last 65.07) - open/close 4506 (time 1776195681.58 total 70.06 last 55.36) - open/close 5129 (time 1776195691.59 total 80.07 last 62.29) - open/close 5743 (time 1776195701.59 total 90.07 last 61.38) - open/close 6333 (time 1776195711.59 total 100.07 last 58.99) - open/close 6894 (time 1776195721.60 total 110.08 last 56.05) - open/close 7496 (time 1776195731.61 total 120.09 last 60.13) - open/close 8140 (time 1776195741.61 total 130.09 last 64.38) - open/close 8804 (time 1776195751.63 total 140.11 last 66.32) - open/close 9434 (time 1776195761.64 total 150.12 last 62.91) total: 10000 open/close in 159.42 seconds: 62.73 ops/second - unlinked 0 (time 1776195774 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 593 (time 1776195886.76 total 10.00 last 59.27) - open/close 1180 (time 1776195896.77 total 20.01 last 58.68) - open/close 1729 (time 1776195906.78 total 30.02 last 54.83) - open/close 2269 (time 1776195916.79 total 40.03 last 53.95) - open/close 2867 (time 1776195926.80 total 50.04 last 59.75) - open/close 3530 (time 1776195936.80 total 60.05 last 66.26) - open/close 4247 (time 1776195946.81 total 70.06 last 71.62) - open/close 4925 (time 1776195956.82 total 80.06 last 67.77) - open/close 5566 (time 1776195966.82 total 90.07 last 64.07) - open/close 6192 (time 1776195976.83 total 100.07 last 62.58) - open/close 6831 (time 1776195986.83 total 110.07 last 63.89) - open/close 7436 (time 1776195996.83 total 120.08 last 60.47) - open/close 8091 (time 1776196006.85 total 130.09 last 65.40) - open/close 8673 (time 1776196016.85 total 140.09 last 58.19) - open/close 9305 (time 1776196026.85 total 150.10 last 63.17) - open/close 9880 (time 1776196036.86 total 160.10 last 57.48) total: 10000 open/close in 162.46 seconds: 61.55 ops/second - unlinked 0 (time 1776196043 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 683 (time 1776196151.03 total 10.00 last 68.29) - open/close 1286 (time 1776196161.04 total 20.01 last 60.25) - open/close 1900 (time 1776196171.05 total 30.01 last 61.37) - open/close 2479 (time 1776196181.05 total 40.02 last 57.85) - open/close 3110 (time 1776196191.07 total 50.03 last 63.01) - open/close 3691 (time 1776196201.07 total 60.04 last 58.05) - open/close 4295 (time 1776196211.08 total 70.04 last 60.40) - open/close 4862 (time 1776196221.09 total 80.05 last 56.64) - open/close 5470 (time 1776196231.09 total 90.06 last 60.77) - open/close 6087 (time 1776196241.10 total 100.07 last 61.63) - open/close 6746 (time 1776196251.12 total 110.09 last 65.78) - open/close 7363 (time 1776196261.13 total 120.10 last 61.63) - open/close 7982 (time 1776196271.13 total 130.10 last 61.88) - open/close 8576 (time 1776196281.14 total 140.11 last 59.37) - open/close 9207 (time 1776196291.15 total 150.12 last 63.01) - open/close 9802 (time 1776196301.18 total 160.15 last 59.32) total: 10000 open/close in 163.38 seconds: 61.21 ops/second - unlinked 0 (time 1776196308 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 610 (time 1776196414.42 total 10.01 last 60.95) - open/close 1247 (time 1776196424.43 total 20.01 last 63.66) - open/close 1842 (time 1776196434.43 total 30.02 last 59.49) - open/close 2416 (time 1776196444.44 total 40.03 last 57.35) - open/close 3063 (time 1776196454.44 total 50.03 last 64.69) - open/close 3655 (time 1776196464.44 total 60.03 last 59.18) - open/close 4266 (time 1776196474.45 total 70.04 last 61.05) - open/close 4917 (time 1776196484.46 total 80.05 last 65.06) - open/close 5523 (time 1776196494.46 total 90.05 last 60.57) - open/close 6126 (time 1776196504.46 total 100.05 last 60.30) - open/close 6717 (time 1776196514.47 total 110.06 last 59.07) - open/close 7361 (time 1776196524.49 total 120.08 last 64.24) - open/close 7959 (time 1776196534.50 total 130.09 last 59.78) - open/close 8596 (time 1776196544.52 total 140.10 last 63.59) - open/close 9144 (time 1776196554.53 total 150.12 last 54.73) - open/close 9712 (time 1776196564.54 total 160.13 last 56.71) total: 10000 open/close in 166.27 seconds: 60.14 ops/second - unlinked 0 (time 1776196575 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 694 (time 1776196682.02 total 10.01 last 69.34) - open/close 1265 (time 1776196692.02 total 20.01 last 57.08) - open/close 1644 (time 1776196702.03 total 30.03 last 37.85) - open/close 2242 (time 1776196712.04 total 40.03 last 59.78) - open/close 2821 (time 1776196722.06 total 50.05 last 57.77) - open/close 3412 (time 1776196732.07 total 60.06 last 59.05) - open/close 4030 (time 1776196742.08 total 70.07 last 61.76) - open/close 4646 (time 1776196752.08 total 80.07 last 61.58) - open/close 5322 (time 1776196762.09 total 90.08 last 67.51) - open/close 5987 (time 1776196772.12 total 100.11 last 66.32) - open/close 6621 (time 1776196782.13 total 110.13 last 63.31) - open/close 7234 (time 1776196792.14 total 120.13 last 61.29) - open/close 7904 (time 1776196802.15 total 130.15 last 66.89) - open/close 8553 (time 1776196812.17 total 140.16 last 64.81) - open/close 9152 (time 1776196822.18 total 150.18 last 59.80) - open/close 9764 (time 1776196832.20 total 160.19 last 61.09) total: 10000 open/close in 164.02 seconds: 60.97 ops/second - unlinked 0 (time 1776196839 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 652 (time 1776196947.41 total 10.02 last 65.09) - open/close 1235 (time 1776196957.42 total 20.03 last 58.25) - open/close 1876 (time 1776196967.43 total 30.04 last 64.04) - open/close 2481 (time 1776196977.44 total 40.04 last 60.47) - open/close 3084 (time 1776196987.46 total 50.06 last 60.18) - open/close 3702 (time 1776196997.49 total 60.10 last 61.58) - open/close 4278 (time 1776197007.51 total 70.11 last 57.54) - open/close 4905 (time 1776197017.51 total 80.12 last 62.65) - open/close 5515 (time 1776197027.52 total 90.12 last 60.97) - open/close 6164 (time 1776197037.53 total 100.13 last 64.86) - open/close 6854 (time 1776197047.53 total 110.13 last 69.00) - open/close 7504 (time 1776197057.54 total 120.14 last 64.91) - open/close 8111 (time 1776197067.54 total 130.14 last 60.70) - open/close 8767 (time 1776197077.57 total 140.17 last 65.42) - open/close 9426 (time 1776197087.58 total 150.19 last 65.80) total: 10000 open/close in 159.28 seconds: 62.78 ops/second - unlinked 0 (time 1776197100 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 653 (time 1776197207.54 total 10.00 last 65.29) - open/close 1238 (time 1776197217.55 total 20.02 last 58.40) - open/close 1884 (time 1776197227.61 total 30.08 last 64.24) - open/close 2502 (time 1776197237.63 total 40.10 last 61.66) - open/close 3105 (time 1776197247.63 total 50.10 last 60.30) - open/close 3749 (time 1776197257.64 total 60.11 last 64.32) - open/close 4370 (time 1776197267.65 total 70.12 last 62.05) - open/close 4959 (time 1776197277.67 total 80.14 last 58.77) - open/close 5673 (time 1776197287.70 total 90.16 last 71.22) - open/close 6269 (time 1776197297.73 total 100.20 last 59.41) - open/close 6908 (time 1776197307.73 total 110.20 last 63.89) - open/close 7517 (time 1776197317.75 total 120.22 last 60.76) - open/close 8139 (time 1776197327.77 total 130.23 last 62.12) - open/close 8780 (time 1776197337.77 total 140.24 last 64.06) - open/close 9340 (time 1776197347.78 total 150.24 last 55.97) - open/close 9941 (time 1776197357.80 total 160.27 last 59.97) total: 10000 open/close in 161.14 seconds: 62.06 ops/second - unlinked 0 (time 1776197363 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 625 (time 1776197476.76 total 10.01 last 62.44) - open/close 1205 (time 1776197486.76 total 20.02 last 57.97) - open/close 1815 (time 1776197496.77 total 30.02 last 60.97) - open/close 2455 (time 1776197506.77 total 40.03 last 63.95) - open/close 3052 (time 1776197516.78 total 50.03 last 59.70) - open/close 3660 (time 1776197526.79 total 60.05 last 60.69) - open/close 4301 (time 1776197536.80 total 70.06 last 64.04) - open/close 4982 (time 1776197546.81 total 80.06 last 68.08) - open/close 5603 (time 1776197556.82 total 90.07 last 62.02) - open/close 6133 (time 1776197566.83 total 100.08 last 52.95) - open/close 6635 (time 1776197576.84 total 110.09 last 50.16) - open/close 7156 (time 1776197586.85 total 120.10 last 52.05) - open/close 7719 (time 1776197596.85 total 130.11 last 56.27) - open/close 8292 (time 1776197606.86 total 140.11 last 57.26) - open/close 8920 (time 1776197616.87 total 150.13 last 62.73) - open/close 9532 (time 1776197626.88 total 160.13 last 61.16) total: 10000 open/close in 167.88 seconds: 59.57 ops/second - unlinked 0 (time 1776197639 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 624 (time 1776197750.29 total 10.01 last 62.31) - open/close 1202 (time 1776197760.30 total 20.02 last 57.76) - open/close 1754 (time 1776197770.31 total 30.03 last 55.15) - open/close 2315 (time 1776197780.32 total 40.05 last 56.02) - open/close 2955 (time 1776197790.33 total 50.05 last 63.94) - open/close 3563 (time 1776197800.34 total 60.06 last 60.74) - open/close 4165 (time 1776197810.36 total 70.09 last 60.07) - open/close 4765 (time 1776197820.37 total 80.10 last 59.94) - open/close 5320 (time 1776197830.38 total 90.10 last 55.45) - open/close 5938 (time 1776197840.39 total 100.12 last 61.73) - open/close 6561 (time 1776197850.41 total 110.13 last 62.22) - open/close 7196 (time 1776197860.41 total 120.14 last 63.45) - open/close 7800 (time 1776197870.42 total 130.15 last 60.34) - open/close 8408 (time 1776197880.43 total 140.15 last 60.76) - open/close 9051 (time 1776197890.43 total 150.15 last 64.29) - open/close 9686 (time 1776197900.45 total 160.17 last 63.40) total: 10000 open/close in 164.93 seconds: 60.63 ops/second - unlinked 0 (time 1776197909 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 722 (time 1776198018.59 total 10.01 last 72.10) - open/close 1315 (time 1776198028.59 total 20.02 last 59.28) - open/close 1909 (time 1776198038.60 total 30.02 last 59.36) - open/close 2499 (time 1776198048.61 total 40.04 last 58.94) - open/close 3097 (time 1776198058.63 total 50.05 last 59.68) - open/close 3731 (time 1776198068.64 total 60.07 last 63.31) - open/close 4360 (time 1776198078.65 total 70.08 last 62.85) - open/close 4947 (time 1776198088.65 total 80.08 last 58.68) - open/close 5550 (time 1776198098.67 total 90.09 last 60.23) - open/close 6123 (time 1776198108.68 total 100.11 last 57.21) - open/close 6759 (time 1776198118.68 total 110.11 last 63.58) - open/close 7344 (time 1776198128.69 total 120.12 last 58.44) - open/close 7972 (time 1776198138.70 total 130.13 last 62.75) - open/close 8624 (time 1776198148.70 total 140.13 last 65.18) - open/close 9242 (time 1776198158.71 total 150.13 last 61.77) - open/close 9860 (time 1776198168.71 total 160.14 last 61.79) total: 10000 open/close in 162.27 seconds: 61.63 ops/second - unlinked 0 (time 1776198175 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 549 (time 1776198288.43 total 10.00 last 54.89) - open/close 1113 (time 1776198298.43 total 20.00 last 56.40) - open/close 1661 (time 1776198308.43 total 30.01 last 54.76) - open/close 2221 (time 1776198318.45 total 40.02 last 55.92) - open/close 2787 (time 1776198328.46 total 50.03 last 56.55) - open/close 3422 (time 1776198338.47 total 60.04 last 63.44) - open/close 4050 (time 1776198348.48 total 70.05 last 62.72) - open/close 4627 (time 1776198358.49 total 80.06 last 57.66) - open/close 5236 (time 1776198368.50 total 90.07 last 60.84) - open/close 5660 (time 1776198378.51 total 100.08 last 42.37) - open/close 6260 (time 1776198388.51 total 110.08 last 60.00) - open/close 6775 (time 1776198398.52 total 120.10 last 51.40) - open/close 7369 (time 1776198408.53 total 130.11 last 59.36) - open/close 8019 (time 1776198418.54 total 140.11 last 64.97) - open/close 8637 (time 1776198428.56 total 150.13 last 61.65) - open/close 9279 (time 1776198438.58 total 160.15 last 64.10) - open/close 9895 (time 1776198448.60 total 170.17 last 61.45) total: 10000 open/close in 171.80 seconds: 58.21 ops/second - unlinked 0 (time 1776198454 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458499 s, 2.3 MB/s fail_loc=0x80000204 Failing ost1 on oleg407-server Stopping /mnt/lustre-ost1 (opts:) on oleg407-server reboot facets: ost1 Failover ost1 to oleg407-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776198643.34 total 33.84 last 0.00) total: 100 open/close in 36.78 seconds: 2.72 ops/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg407-server: LNET ready to unload oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.