== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:14:02 (1781280842) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg420-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 614 (time 1781281028.31 total 10.00 last 61.38) - open/close 1311 (time 1781281038.32 total 20.01 last 69.68) - open/close 1937 (time 1781281048.34 total 30.03 last 62.47) - open/close 2559 (time 1781281058.35 total 40.04 last 62.13) - open/close 3192 (time 1781281068.38 total 50.07 last 63.10) - open/close 3827 (time 1781281078.39 total 60.08 last 63.45) - open/close 4447 (time 1781281088.40 total 70.09 last 61.91) - open/close 5118 (time 1781281098.41 total 80.10 last 67.06) - open/close 5703 (time 1781281108.42 total 90.11 last 58.41) - open/close 6273 (time 1781281118.44 total 100.13 last 56.90) - open/close 6908 (time 1781281128.44 total 110.14 last 63.48) - open/close 7563 (time 1781281138.46 total 120.15 last 65.43) - open/close 8231 (time 1781281148.46 total 130.15 last 66.77) - open/close 8888 (time 1781281158.47 total 140.16 last 65.64) - open/close 9481 (time 1781281168.47 total 150.16 last 59.30) total: 10000 open/close in 159.22 seconds: 62.80 ops/second - unlinked 0 (time 1781281182 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 664 (time 1781281296.40 total 10.01 last 66.32) - open/close 1264 (time 1781281306.43 total 20.04 last 59.85) - open/close 1882 (time 1781281316.43 total 30.04 last 61.80) - open/close 2565 (time 1781281326.44 total 40.05 last 68.23) - open/close 3255 (time 1781281336.45 total 50.06 last 68.94) - open/close 3861 (time 1781281346.48 total 60.09 last 60.42) - open/close 4474 (time 1781281356.49 total 70.10 last 61.20) - open/close 5091 (time 1781281366.49 total 80.10 last 61.70) - open/close 5711 (time 1781281376.51 total 90.12 last 61.90) - open/close 6325 (time 1781281386.52 total 100.13 last 61.34) - open/close 6995 (time 1781281396.53 total 110.14 last 66.96) - open/close 7597 (time 1781281406.53 total 120.14 last 60.19) - open/close 8182 (time 1781281416.54 total 130.15 last 58.45) - open/close 8780 (time 1781281426.54 total 140.15 last 59.76) - open/close 9363 (time 1781281436.55 total 150.16 last 58.25) - open/close 9990 (time 1781281446.56 total 160.17 last 62.66) total: 10000 open/close in 160.30 seconds: 62.38 ops/second - unlinked 0 (time 1781281450 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 674 (time 1781281559.24 total 10.00 last 67.38) - open/close 1395 (time 1781281569.24 total 20.01 last 72.05) - open/close 2044 (time 1781281579.25 total 30.02 last 64.83) - open/close 2743 (time 1781281589.25 total 40.02 last 69.89) - open/close 3421 (time 1781281599.26 total 50.03 last 67.74) - open/close 4128 (time 1781281609.27 total 60.03 last 70.69) - open/close 4769 (time 1781281619.27 total 70.04 last 64.07) - open/close 5413 (time 1781281629.28 total 80.05 last 64.31) - open/close 5958 (time 1781281639.29 total 90.05 last 54.49) - open/close 6569 (time 1781281649.29 total 100.06 last 61.05) - open/close 7163 (time 1781281659.30 total 110.07 last 59.36) - open/close 7739 (time 1781281669.31 total 120.07 last 57.57) - open/close 8382 (time 1781281679.31 total 130.08 last 64.25) - open/close 9023 (time 1781281689.32 total 140.09 last 64.07) - open/close 9710 (time 1781281699.32 total 150.09 last 68.69) total: 10000 open/close in 154.18 seconds: 64.86 ops/second - unlinked 0 (time 1781281707 ; total 1 ; last 1) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 500 (time 1781281810.81 total 10.01 last 49.93) - open/close 1102 (time 1781281820.83 total 20.03 last 60.10) - open/close 1763 (time 1781281830.85 total 30.05 last 65.97) - open/close 2439 (time 1781281840.86 total 40.06 last 67.55) - open/close 3127 (time 1781281850.87 total 50.07 last 68.71) - open/close 3836 (time 1781281860.88 total 60.08 last 70.85) - open/close 4355 (time 1781281870.88 total 70.08 last 51.89) - open/close 4979 (time 1781281880.88 total 80.08 last 62.39) - open/close 5681 (time 1781281890.88 total 90.09 last 70.17) - open/close 6349 (time 1781281900.90 total 100.10 last 66.70) - open/close 7043 (time 1781281910.91 total 110.11 last 69.34) - open/close 7743 (time 1781281920.92 total 120.12 last 69.90) - open/close 8420 (time 1781281930.93 total 130.13 last 67.62) - open/close 9061 (time 1781281940.95 total 140.15 last 64.03) - open/close 9667 (time 1781281950.96 total 150.16 last 60.53) total: 10000 open/close in 155.88 seconds: 64.15 ops/second - unlinked 0 (time 1781281961 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 631 (time 1781282068.64 total 10.01 last 63.03) - open/close 1271 (time 1781282078.65 total 20.02 last 63.97) - open/close 1944 (time 1781282088.65 total 30.02 last 67.29) - open/close 2602 (time 1781282098.66 total 40.03 last 65.73) - open/close 3188 (time 1781282108.66 total 50.03 last 58.57) - open/close 3763 (time 1781282118.67 total 60.04 last 57.47) - open/close 4379 (time 1781282128.67 total 70.04 last 61.58) - open/close 5024 (time 1781282138.68 total 80.05 last 64.46) - open/close 5673 (time 1781282148.69 total 90.06 last 64.84) - open/close 6302 (time 1781282158.71 total 100.08 last 62.78) - open/close 7010 (time 1781282168.72 total 110.09 last 70.69) - open/close 7649 (time 1781282178.74 total 120.11 last 63.80) - open/close 8349 (time 1781282188.74 total 130.11 last 69.99) - open/close 9002 (time 1781282198.75 total 140.11 last 65.26) - open/close 9675 (time 1781282208.75 total 150.12 last 67.24) total: 10000 open/close in 154.68 seconds: 64.65 ops/second - unlinked 0 (time 1781282217 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 602 (time 1781282322.89 total 10.01 last 60.13) - open/close 1194 (time 1781282332.89 total 20.01 last 59.20) - open/close 1838 (time 1781282342.89 total 30.02 last 64.35) - open/close 2513 (time 1781282352.90 total 40.03 last 67.44) - open/close 3183 (time 1781282362.91 total 50.03 last 66.98) - open/close 3878 (time 1781282372.92 total 60.05 last 69.41) - open/close 4513 (time 1781282382.93 total 70.06 last 63.42) - open/close 5143 (time 1781282392.95 total 80.08 last 62.87) - open/close 5710 (time 1781282402.96 total 90.09 last 56.66) - open/close 6319 (time 1781282412.96 total 100.09 last 60.90) - open/close 7004 (time 1781282422.97 total 110.09 last 68.45) - open/close 7537 (time 1781282432.98 total 120.10 last 53.24) - open/close 8158 (time 1781282442.98 total 130.11 last 62.06) - open/close 8754 (time 1781282452.98 total 140.11 last 59.59) - open/close 9429 (time 1781282463.01 total 150.14 last 67.34) total: 10000 open/close in 159.30 seconds: 62.78 ops/second - unlinked 0 (time 1781282476 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 734 (time 1781282584.22 total 10.02 last 73.27) - open/close 1450 (time 1781282594.22 total 20.02 last 71.58) - open/close 2100 (time 1781282604.24 total 30.04 last 64.88) - open/close 2769 (time 1781282614.25 total 40.04 last 66.86) - open/close 3451 (time 1781282624.25 total 50.05 last 68.17) - open/close 4114 (time 1781282634.26 total 60.06 last 66.25) - open/close 4776 (time 1781282644.26 total 70.06 last 66.17) - open/close 5398 (time 1781282654.27 total 80.07 last 62.16) - open/close 6070 (time 1781282664.28 total 90.07 last 67.16) - open/close 6653 (time 1781282674.29 total 100.09 last 58.21) - open/close 7280 (time 1781282684.29 total 110.09 last 62.69) - open/close 7993 (time 1781282694.29 total 120.09 last 71.28) - open/close 8611 (time 1781282704.31 total 130.11 last 61.69) - open/close 9261 (time 1781282714.32 total 140.12 last 64.96) - open/close 9887 (time 1781282724.32 total 150.12 last 62.56) total: 10000 open/close in 151.72 seconds: 65.91 ops/second - unlinked 0 (time 1781282730 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 703 (time 1781282831.89 total 10.01 last 70.23) - open/close 1362 (time 1781282841.89 total 20.01 last 65.89) - open/close 2032 (time 1781282851.89 total 30.01 last 67.00) - open/close 2661 (time 1781282861.89 total 40.01 last 62.88) - open/close 3393 (time 1781282871.90 total 50.02 last 73.13) - open/close 4124 (time 1781282881.91 total 60.03 last 73.04) - open/close 4874 (time 1781282891.92 total 70.04 last 74.96) - open/close 5560 (time 1781282901.92 total 80.04 last 68.58) - open/close 6235 (time 1781282911.94 total 90.05 last 67.39) - open/close 6830 (time 1781282921.95 total 100.07 last 59.42) - open/close 7485 (time 1781282931.96 total 110.08 last 65.42) - open/close 8096 (time 1781282941.97 total 120.09 last 61.03) - open/close 8706 (time 1781282951.98 total 130.10 last 60.94) - open/close 9335 (time 1781282961.99 total 140.10 last 62.87) - open/close 9973 (time 1781282971.99 total 150.10 last 63.80) total: 10000 open/close in 150.52 seconds: 66.44 ops/second - unlinked 0 (time 1781282977 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 628 (time 1781283077.81 total 10.01 last 62.76) - open/close 1185 (time 1781283087.81 total 20.01 last 55.68) - open/close 1861 (time 1781283097.84 total 30.03 last 67.45) - open/close 2570 (time 1781283107.84 total 40.03 last 70.89) - open/close 3218 (time 1781283117.84 total 50.04 last 64.76) - open/close 3948 (time 1781283127.85 total 60.04 last 72.99) - open/close 4689 (time 1781283137.85 total 70.05 last 74.04) - open/close 5482 (time 1781283147.86 total 80.06 last 79.24) - open/close 6222 (time 1781283157.89 total 90.08 last 73.80) - open/close 6901 (time 1781283167.92 total 100.11 last 67.70) - open/close 7554 (time 1781283177.92 total 110.11 last 65.28) - open/close 8198 (time 1781283187.92 total 120.12 last 64.36) - open/close 8839 (time 1781283197.94 total 130.14 last 64.00) - open/close 9518 (time 1781283207.95 total 140.14 last 67.86) total: 10000 open/close in 147.23 seconds: 67.92 ops/second - unlinked 0 (time 1781283218 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 655 (time 1781283316.58 total 10.01 last 65.45) - open/close 1343 (time 1781283326.61 total 20.03 last 68.61) - open/close 1996 (time 1781283336.62 total 30.04 last 65.24) - open/close 2649 (time 1781283346.62 total 40.05 last 65.27) - open/close 3306 (time 1781283356.63 total 50.06 last 65.62) - open/close 3896 (time 1781283366.64 total 60.06 last 58.97) - open/close 4537 (time 1781283376.65 total 70.08 last 64.01) - open/close 5198 (time 1781283386.66 total 80.09 last 66.05) - open/close 5854 (time 1781283396.68 total 90.11 last 65.44) - open/close 6495 (time 1781283406.69 total 100.12 last 64.04) - open/close 7156 (time 1781283416.71 total 110.13 last 66.00) - open/close 7798 (time 1781283426.71 total 120.14 last 64.14) - open/close 8432 (time 1781283436.73 total 130.15 last 63.33) - open/close 9069 (time 1781283446.73 total 140.16 last 63.65) - open/close 9742 (time 1781283456.74 total 150.17 last 67.27) total: 10000 open/close in 154.01 seconds: 64.93 ops/second - unlinked 0 (time 1781283464 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 674 (time 1781283574.47 total 10.00 last 67.37) - open/close 1418 (time 1781283584.48 total 20.02 last 74.31) - open/close 2108 (time 1781283594.49 total 30.03 last 68.92) - open/close 2801 (time 1781283604.49 total 40.03 last 69.29) - open/close 3492 (time 1781283614.50 total 50.04 last 69.04) - open/close 4165 (time 1781283624.51 total 60.05 last 67.21) - open/close 4796 (time 1781283634.52 total 70.06 last 63.04) - open/close 5429 (time 1781283644.53 total 80.07 last 63.27) - open/close 6018 (time 1781283654.54 total 90.08 last 58.85) - open/close 6596 (time 1781283664.55 total 100.09 last 57.74) - open/close 7217 (time 1781283674.55 total 110.09 last 62.10) - open/close 7844 (time 1781283684.56 total 120.10 last 62.63) - open/close 8514 (time 1781283694.58 total 130.12 last 66.86) - open/close 9153 (time 1781283704.59 total 140.12 last 63.85) - open/close 9813 (time 1781283714.60 total 150.14 last 65.92) total: 10000 open/close in 152.90 seconds: 65.40 ops/second - unlinked 0 (time 1781283721 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4224 377344 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4831 s, 2.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg420-server Stopping /mnt/lustre-ost1 (opts:) on oleg420-server reboot facets: ost1 Failover ost1 to oleg420-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1781283913.01 total 36.07 last 0.00) total: 100 open/close in 38.76 seconds: 2.58 ops/second umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg420-server: LNET ready to unload oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.