== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 13:49:59 (1776188999) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock oleg245-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 843 (time 1776189118.92 total 10.01 last 84.24) - open/close 1735 (time 1776189128.93 total 20.02 last 89.12) - open/close 2516 (time 1776189138.95 total 30.03 last 77.97) - open/close 3338 (time 1776189148.96 total 40.04 last 82.14) - open/close 4003 (time 1776189158.97 total 50.05 last 66.44) - open/close 4767 (time 1776189168.99 total 60.07 last 76.21) - open/close 5447 (time 1776189179.00 total 70.08 last 67.93) - open/close 6283 (time 1776189189.01 total 80.09 last 83.54) - open/close 7075 (time 1776189199.01 total 90.09 last 79.18) - open/close 7840 (time 1776189209.02 total 100.11 last 76.42) - open/close 8663 (time 1776189219.03 total 110.11 last 82.24) - open/close 9490 (time 1776189229.03 total 120.11 last 82.68) total: 10000 open/close in 126.61 seconds: 78.98 ops/second - unlinked 0 (time 1776189239 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 862 (time 1776189330.58 total 10.01 last 86.12) - open/close 1588 (time 1776189340.59 total 20.01 last 72.57) - open/close 2275 (time 1776189350.60 total 30.02 last 68.64) - open/close 2908 (time 1776189360.62 total 40.04 last 63.17) - open/close 3587 (time 1776189370.62 total 50.05 last 67.85) - open/close 4234 (time 1776189380.63 total 60.05 last 64.69) - open/close 5015 (time 1776189390.63 total 70.05 last 78.07) - open/close 5789 (time 1776189400.67 total 80.09 last 77.12) - open/close 6578 (time 1776189410.67 total 90.10 last 78.87) - open/close 7326 (time 1776189420.68 total 100.10 last 74.76) - open/close 8036 (time 1776189430.69 total 110.11 last 70.92) - open/close 8754 (time 1776189440.70 total 120.12 last 71.73) - open/close 9472 (time 1776189450.71 total 130.14 last 71.68) total: 10000 open/close in 137.95 seconds: 72.49 ops/second - unlinked 0 (time 1776189463 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 617 (time 1776189570.43 total 10.02 last 61.56) - open/close 1283 (time 1776189580.44 total 20.03 last 66.57) - open/close 1837 (time 1776189590.46 total 30.05 last 55.29) - open/close 2472 (time 1776189600.49 total 40.08 last 63.31) - open/close 3172 (time 1776189610.50 total 50.09 last 69.93) - open/close 3909 (time 1776189620.50 total 60.09 last 73.69) - open/close 4715 (time 1776189630.50 total 70.10 last 80.54) - open/close 5487 (time 1776189640.51 total 80.10 last 77.16) - open/close 6229 (time 1776189650.52 total 90.11 last 74.11) - open/close 6942 (time 1776189660.52 total 100.11 last 71.29) - open/close 7520 (time 1776189670.52 total 110.12 last 57.79) - open/close 8111 (time 1776189680.53 total 120.13 last 59.04) - open/close 8669 (time 1776189690.54 total 130.13 last 55.77) - open/close 9307 (time 1776189700.54 total 140.13 last 63.79) - open/close 9932 (time 1776189710.55 total 150.14 last 62.47) total: 10000 open/close in 151.14 seconds: 66.17 ops/second - unlinked 0 (time 1776189716 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 621 (time 1776189819.04 total 10.01 last 62.02) - open/close 1216 (time 1776189829.06 total 20.02 last 59.43) - open/close 1834 (time 1776189839.09 total 30.06 last 61.59) - open/close 2338 (time 1776189849.10 total 40.06 last 50.37) - open/close 2885 (time 1776189859.11 total 50.08 last 54.62) - open/close 3547 (time 1776189869.12 total 60.09 last 66.11) - open/close 4285 (time 1776189879.13 total 70.10 last 73.77) - open/close 5031 (time 1776189889.13 total 80.10 last 74.59) - open/close 5717 (time 1776189899.14 total 90.10 last 68.57) - open/close 6352 (time 1776189909.14 total 100.11 last 63.49) - open/close 6999 (time 1776189919.14 total 110.11 last 64.70) - open/close 7594 (time 1776189929.15 total 120.12 last 59.44) - open/close 8344 (time 1776189939.15 total 130.12 last 74.97) - open/close 9075 (time 1776189949.15 total 140.12 last 73.09) - open/close 9792 (time 1776189959.16 total 150.13 last 71.67) total: 10000 open/close in 152.54 seconds: 65.56 ops/second - unlinked 0 (time 1776189966 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 748 (time 1776190063.18 total 10.01 last 74.70) - open/close 1466 (time 1776190073.18 total 20.02 last 71.76) - open/close 2191 (time 1776190083.18 total 30.02 last 72.48) - open/close 2877 (time 1776190093.20 total 40.03 last 68.51) - open/close 3560 (time 1776190103.21 total 50.04 last 68.25) - open/close 4238 (time 1776190113.21 total 60.05 last 67.77) - open/close 4985 (time 1776190123.22 total 70.05 last 74.63) - open/close 5727 (time 1776190133.23 total 80.06 last 74.13) - open/close 6387 (time 1776190143.24 total 90.08 last 65.92) - open/close 7149 (time 1776190153.29 total 100.12 last 75.85) - open/close 7809 (time 1776190163.30 total 110.14 last 65.91) - open/close 8475 (time 1776190173.31 total 120.14 last 66.57) - open/close 9162 (time 1776190183.32 total 130.16 last 68.58) - open/close 9890 (time 1776190193.32 total 140.16 last 72.79) total: 10000 open/close in 142.17 seconds: 70.34 ops/second - unlinked 0 (time 1776190201 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 769 (time 1776190303.27 total 10.01 last 76.86) - open/close 1417 (time 1776190313.28 total 20.02 last 64.73) - open/close 2170 (time 1776190323.28 total 30.02 last 75.28) - open/close 2835 (time 1776190333.28 total 40.02 last 66.47) - open/close 3597 (time 1776190343.28 total 50.02 last 76.20) - open/close 4216 (time 1776190353.29 total 60.03 last 61.89) - open/close 4863 (time 1776190363.29 total 70.03 last 64.68) - open/close 5565 (time 1776190373.29 total 80.03 last 70.18) - open/close 6422 (time 1776190383.30 total 90.04 last 85.60) - open/close 7138 (time 1776190393.31 total 100.05 last 71.57) - open/close 7839 (time 1776190403.31 total 110.05 last 70.05) - open/close 8662 (time 1776190413.32 total 120.06 last 82.24) - open/close 9416 (time 1776190423.33 total 130.07 last 75.36) total: 10000 open/close in 138.64 seconds: 72.13 ops/second - unlinked 0 (time 1776190437 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 697 (time 1776190536.82 total 10.00 last 69.67) - open/close 1398 (time 1776190546.83 total 20.01 last 70.04) - open/close 2127 (time 1776190556.84 total 30.02 last 72.84) - open/close 2834 (time 1776190566.85 total 40.03 last 70.61) - open/close 3480 (time 1776190576.86 total 50.04 last 64.56) - open/close 4216 (time 1776190586.86 total 60.04 last 73.58) - open/close 4998 (time 1776190596.87 total 70.05 last 78.11) - open/close 5701 (time 1776190606.88 total 80.06 last 70.29) - open/close 6520 (time 1776190616.88 total 90.06 last 81.88) - open/close 7223 (time 1776190626.90 total 100.07 last 70.19) - open/close 7743 (time 1776190636.90 total 110.08 last 51.98) - open/close 8442 (time 1776190646.90 total 120.08 last 69.89) - open/close 9183 (time 1776190656.91 total 130.09 last 74.02) - open/close 9855 (time 1776190666.92 total 140.10 last 67.18) total: 10000 open/close in 143.10 seconds: 69.88 ops/second - unlinked 0 (time 1776190675 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 657 (time 1776190782.02 total 10.00 last 65.69) - open/close 1207 (time 1776190792.04 total 20.02 last 54.91) - open/close 1864 (time 1776190802.05 total 30.03 last 65.63) - open/close 2568 (time 1776190812.05 total 40.03 last 70.38) - open/close 3342 (time 1776190822.05 total 50.03 last 77.40) - open/close 4091 (time 1776190832.06 total 60.04 last 74.85) - open/close 4833 (time 1776190842.06 total 70.04 last 74.18) - open/close 5494 (time 1776190852.06 total 80.04 last 66.09) - open/close 6271 (time 1776190862.08 total 90.06 last 77.55) - open/close 7049 (time 1776190872.09 total 100.07 last 77.75) - open/close 7659 (time 1776190882.09 total 110.07 last 60.98) - open/close 8223 (time 1776190892.10 total 120.08 last 56.37) - open/close 8725 (time 1776190902.11 total 130.09 last 50.12) - open/close 9251 (time 1776190912.14 total 140.12 last 52.47) - open/close 9812 (time 1776190922.14 total 150.12 last 56.10) total: 10000 open/close in 152.98 seconds: 65.37 ops/second - unlinked 0 (time 1776190929 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 610 (time 1776191035.19 total 10.00 last 60.99) - open/close 1334 (time 1776191045.19 total 20.00 last 72.40) - open/close 1907 (time 1776191055.20 total 30.02 last 57.22) - open/close 2406 (time 1776191065.21 total 40.02 last 49.88) - open/close 2957 (time 1776191075.22 total 50.03 last 55.06) - open/close 3613 (time 1776191085.25 total 60.07 last 65.34) - open/close 4202 (time 1776191095.27 total 70.08 last 58.83) - open/close 4985 (time 1776191105.27 total 80.08 last 78.26) - open/close 5750 (time 1776191115.28 total 90.09 last 76.47) - open/close 6478 (time 1776191125.30 total 100.11 last 72.65) - open/close 7065 (time 1776191135.30 total 110.11 last 58.67) - open/close 7796 (time 1776191145.30 total 120.11 last 73.10) - open/close 8473 (time 1776191155.31 total 130.12 last 67.63) - open/close 9229 (time 1776191165.32 total 140.13 last 75.56) - open/close 9968 (time 1776191175.34 total 150.15 last 73.75) total: 10000 open/close in 150.74 seconds: 66.34 ops/second - unlinked 0 (time 1776191181 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 713 (time 1776191279.20 total 10.00 last 71.29) - open/close 1499 (time 1776191289.21 total 20.01 last 78.53) - open/close 2243 (time 1776191299.21 total 30.01 last 74.38) - open/close 3011 (time 1776191309.22 total 40.02 last 76.72) - open/close 3721 (time 1776191319.23 total 50.03 last 70.97) - open/close 4475 (time 1776191329.24 total 60.04 last 75.30) - open/close 5195 (time 1776191339.25 total 70.05 last 71.92) - open/close 5919 (time 1776191349.25 total 80.06 last 72.38) - open/close 6528 (time 1776191359.26 total 90.07 last 60.84) - open/close 7096 (time 1776191369.27 total 100.07 last 56.76) - open/close 7809 (time 1776191379.28 total 110.08 last 71.27) - open/close 8547 (time 1776191389.28 total 120.08 last 73.76) - open/close 9283 (time 1776191399.28 total 130.08 last 73.60) total: 10000 open/close in 139.21 seconds: 71.83 ops/second - unlinked 0 (time 1776191413 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 841 (time 1776191513.63 total 10.00 last 84.10) - open/close 1721 (time 1776191523.63 total 20.01 last 87.94) - open/close 2387 (time 1776191533.64 total 30.01 last 66.56) - open/close 3011 (time 1776191543.66 total 40.03 last 62.28) - open/close 3669 (time 1776191553.66 total 50.04 last 65.78) - open/close 4309 (time 1776191563.66 total 60.04 last 63.99) - open/close 5015 (time 1776191573.67 total 70.04 last 70.55) - open/close 5680 (time 1776191583.68 total 80.05 last 66.47) - open/close 6382 (time 1776191593.68 total 90.06 last 70.16) - open/close 7179 (time 1776191603.69 total 100.06 last 79.66) - open/close 7968 (time 1776191613.69 total 110.06 last 78.90) - open/close 8809 (time 1776191623.69 total 120.07 last 84.03) - open/close 9436 (time 1776191633.73 total 130.10 last 62.48) total: 10000 open/close in 138.87 seconds: 72.01 ops/second - unlinked 0 (time 1776191648 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.530954 s, 2.0 MB/s fail_loc=0x80000204 Failing ost1 on oleg245-server Stopping /mnt/lustre-ost1 (opts:) on oleg245-server reboot facets: ost1 Failover ost1 to oleg245-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776191819.48 total 36.03 last 0.00) total: 100 open/close in 38.64 seconds: 2.59 ops/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg245-server: LNET ready to unload oleg245-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.