== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 16:34:50 (1776198890) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg430-server: detected 4 online CPUs by sysfs oleg430-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg430-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock oleg430-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 637 (time 1776199034.10 total 10.03 last 63.51) - open/close 1267 (time 1776199044.10 total 20.03 last 63.00) - open/close 1910 (time 1776199054.10 total 30.04 last 64.25) - open/close 2566 (time 1776199064.13 total 40.06 last 65.46) - open/close 3223 (time 1776199074.14 total 50.08 last 65.60) - open/close 3820 (time 1776199084.14 total 60.08 last 59.69) - open/close 4404 (time 1776199094.20 total 70.13 last 58.08) - open/close 4906 (time 1776199104.22 total 80.15 last 50.10) - open/close 5446 (time 1776199114.23 total 90.17 last 53.92) - open/close 6056 (time 1776199124.24 total 100.17 last 60.96) - open/close 6657 (time 1776199134.24 total 110.18 last 60.07) - open/close 7277 (time 1776199144.25 total 120.19 last 61.95) - open/close 7889 (time 1776199154.26 total 130.20 last 61.15) - open/close 8453 (time 1776199164.27 total 140.20 last 56.35) - open/close 9006 (time 1776199174.28 total 150.21 last 55.27) - open/close 9621 (time 1776199184.28 total 160.22 last 61.46) total: 10000 open/close in 167.58 seconds: 59.67 ops/second - unlinked 0 (time 1776199197 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 619 (time 1776199308.87 total 10.01 last 61.82) - open/close 1250 (time 1776199318.87 total 20.01 last 63.10) - open/close 1907 (time 1776199328.87 total 30.01 last 65.69) - open/close 2506 (time 1776199338.89 total 40.03 last 59.83) - open/close 3116 (time 1776199348.90 total 50.04 last 60.91) - open/close 3745 (time 1776199358.92 total 60.06 last 62.80) - open/close 4320 (time 1776199368.93 total 70.07 last 57.44) - open/close 4878 (time 1776199378.94 total 80.08 last 55.75) - open/close 5506 (time 1776199388.95 total 90.09 last 62.73) - open/close 6132 (time 1776199398.96 total 100.10 last 62.53) - open/close 6665 (time 1776199408.98 total 110.12 last 53.19) - open/close 7254 (time 1776199418.99 total 120.13 last 58.86) - open/close 7795 (time 1776199429.00 total 130.14 last 54.06) - open/close 8413 (time 1776199439.00 total 140.14 last 61.77) - open/close 9076 (time 1776199449.01 total 150.15 last 66.23) - open/close 9725 (time 1776199459.01 total 160.15 last 64.88) total: 10000 open/close in 164.83 seconds: 60.67 ops/second - unlinked 0 (time 1776199469 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 501 (time 1776199582.32 total 10.01 last 50.06) - open/close 1069 (time 1776199592.33 total 20.02 last 56.75) - open/close 1710 (time 1776199602.36 total 30.05 last 63.91) - open/close 2266 (time 1776199612.37 total 40.06 last 55.54) - open/close 2906 (time 1776199622.37 total 50.06 last 64.00) - open/close 3561 (time 1776199632.38 total 60.07 last 65.41) - open/close 4214 (time 1776199642.38 total 70.07 last 65.30) - open/close 4869 (time 1776199652.39 total 80.08 last 65.43) - open/close 5537 (time 1776199662.39 total 90.09 last 66.78) - open/close 6198 (time 1776199672.40 total 100.09 last 66.09) - open/close 6802 (time 1776199682.40 total 110.10 last 60.36) - open/close 7427 (time 1776199692.41 total 120.11 last 62.43) - open/close 8100 (time 1776199702.42 total 130.11 last 67.27) - open/close 8726 (time 1776199712.45 total 140.14 last 62.39) - open/close 9274 (time 1776199722.46 total 150.15 last 54.78) - open/close 9906 (time 1776199732.46 total 160.16 last 63.15) total: 10000 open/close in 161.66 seconds: 61.86 ops/second - unlinked 0 (time 1776199738 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 604 (time 1776199848.74 total 10.01 last 60.33) - open/close 1275 (time 1776199858.75 total 20.02 last 67.03) - open/close 1876 (time 1776199868.75 total 30.03 last 60.05) - open/close 2520 (time 1776199878.78 total 40.06 last 64.22) - open/close 3151 (time 1776199888.78 total 50.06 last 63.09) - open/close 3763 (time 1776199898.79 total 60.06 last 61.18) - open/close 4409 (time 1776199908.79 total 70.07 last 64.58) - open/close 5007 (time 1776199918.79 total 80.07 last 59.79) - open/close 5587 (time 1776199928.80 total 90.08 last 57.95) - open/close 6195 (time 1776199938.83 total 100.10 last 60.63) - open/close 6802 (time 1776199948.83 total 110.10 last 60.70) - open/close 7456 (time 1776199958.84 total 120.12 last 65.32) - open/close 8080 (time 1776199968.84 total 130.12 last 62.37) - open/close 8663 (time 1776199978.86 total 140.13 last 58.22) - open/close 9230 (time 1776199988.89 total 150.16 last 56.53) - open/close 9845 (time 1776199998.90 total 160.18 last 61.42) total: 10000 open/close in 162.82 seconds: 61.42 ops/second - unlinked 0 (time 1776200007 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 601 (time 1776200119.08 total 10.01 last 60.07) - open/close 1222 (time 1776200129.09 total 20.01 last 62.04) - open/close 1853 (time 1776200139.10 total 30.02 last 63.08) - open/close 2452 (time 1776200149.10 total 40.02 last 59.87) - open/close 3039 (time 1776200159.11 total 50.03 last 58.65) - open/close 3643 (time 1776200169.12 total 60.04 last 60.37) - open/close 4186 (time 1776200179.12 total 70.04 last 54.28) - open/close 4605 (time 1776200189.13 total 80.05 last 41.87) - open/close 5011 (time 1776200199.13 total 90.05 last 40.59) - open/close 5585 (time 1776200209.14 total 100.06 last 57.36) - open/close 6128 (time 1776200219.14 total 110.06 last 54.28) - open/close 6743 (time 1776200229.15 total 120.07 last 61.47) - open/close 7364 (time 1776200239.15 total 130.07 last 62.07) - open/close 7991 (time 1776200249.15 total 140.07 last 62.70) - open/close 8571 (time 1776200259.17 total 150.09 last 57.91) - open/close 9120 (time 1776200269.18 total 160.10 last 54.84) - open/close 9772 (time 1776200279.19 total 170.12 last 65.09) total: 10000 open/close in 174.36 seconds: 57.35 ops/second - unlinked 0 (time 1776200287 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 500 (time 1776200391.32 total 10.01 last 49.96) - open/close 1064 (time 1776200401.32 total 20.01 last 56.39) - open/close 1762 (time 1776200411.34 total 30.02 last 69.71) - open/close 2432 (time 1776200421.35 total 40.04 last 66.91) - open/close 3092 (time 1776200431.36 total 50.05 last 65.93) - open/close 3710 (time 1776200441.37 total 60.05 last 61.75) - open/close 4348 (time 1776200451.38 total 70.07 last 63.73) - open/close 4939 (time 1776200461.38 total 80.07 last 59.08) - open/close 5484 (time 1776200471.38 total 90.07 last 54.49) - open/close 5962 (time 1776200481.39 total 100.08 last 47.75) - open/close 6438 (time 1776200491.39 total 110.08 last 47.59) - open/close 6832 (time 1776200501.40 total 120.08 last 39.40) - open/close 7329 (time 1776200511.43 total 130.12 last 49.53) - open/close 7921 (time 1776200521.49 total 140.18 last 58.85) - open/close 8551 (time 1776200531.50 total 150.19 last 62.93) - open/close 9191 (time 1776200541.51 total 160.20 last 63.92) - open/close 9822 (time 1776200551.52 total 170.21 last 63.03) total: 10000 open/close in 173.21 seconds: 57.73 ops/second - unlinked 0 (time 1776200559 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 631 (time 1776200667.91 total 10.01 last 63.05) - open/close 1256 (time 1776200677.94 total 20.03 last 62.35) - open/close 1891 (time 1776200687.95 total 30.04 last 63.44) - open/close 2447 (time 1776200697.96 total 40.05 last 55.53) - open/close 3015 (time 1776200707.96 total 50.06 last 56.79) - open/close 3577 (time 1776200717.97 total 60.06 last 56.17) - open/close 4204 (time 1776200727.97 total 70.07 last 62.66) - open/close 4815 (time 1776200737.98 total 80.07 last 61.07) - open/close 5489 (time 1776200747.99 total 90.08 last 67.33) - open/close 6138 (time 1776200758.00 total 100.09 last 64.84) - open/close 6705 (time 1776200768.00 total 110.10 last 56.69) - open/close 7098 (time 1776200778.02 total 120.12 last 39.21) - open/close 7541 (time 1776200788.03 total 130.12 last 44.27) - open/close 8043 (time 1776200798.04 total 140.14 last 50.13) - open/close 8518 (time 1776200808.05 total 150.14 last 47.48) - open/close 8981 (time 1776200818.05 total 160.15 last 46.28) - open/close 9617 (time 1776200828.07 total 170.17 last 63.49) total: 10000 open/close in 176.84 seconds: 56.55 ops/second - unlinked 0 (time 1776200839 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 655 (time 1776200949.55 total 10.00 last 65.47) - open/close 1265 (time 1776200959.56 total 20.01 last 60.95) - open/close 1906 (time 1776200969.57 total 30.02 last 64.05) - open/close 2495 (time 1776200979.57 total 40.02 last 58.89) - open/close 3086 (time 1776200989.57 total 50.03 last 59.08) - open/close 3682 (time 1776200999.58 total 60.03 last 59.56) - open/close 4374 (time 1776201009.59 total 70.04 last 69.16) - open/close 5033 (time 1776201019.60 total 80.05 last 65.83) - open/close 5686 (time 1776201029.60 total 90.05 last 65.25) - open/close 6292 (time 1776201039.61 total 100.06 last 60.59) - open/close 7009 (time 1776201049.61 total 110.06 last 71.69) - open/close 7656 (time 1776201059.63 total 120.08 last 64.57) - open/close 8342 (time 1776201069.64 total 130.09 last 68.51) - open/close 8965 (time 1776201079.64 total 140.09 last 62.29) - open/close 9636 (time 1776201089.64 total 150.09 last 67.08) total: 10000 open/close in 156.94 seconds: 63.72 ops/second - unlinked 0 (time 1776201100 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 672 (time 1776201207.99 total 10.01 last 67.16) - open/close 1256 (time 1776201218.00 total 20.02 last 58.34) - open/close 1944 (time 1776201228.00 total 30.02 last 68.79) - open/close 2537 (time 1776201238.01 total 40.03 last 59.23) - open/close 3226 (time 1776201248.02 total 50.03 last 68.86) - open/close 3854 (time 1776201258.02 total 60.04 last 62.78) - open/close 4441 (time 1776201268.03 total 70.05 last 58.65) - open/close 5069 (time 1776201278.04 total 80.06 last 62.73) - open/close 5695 (time 1776201288.04 total 90.06 last 62.59) - open/close 6378 (time 1776201298.05 total 100.07 last 68.26) - open/close 7026 (time 1776201308.07 total 110.08 last 64.68) - open/close 7677 (time 1776201318.07 total 120.09 last 65.06) - open/close 8338 (time 1776201328.08 total 130.10 last 66.07) - open/close 9009 (time 1776201338.08 total 140.10 last 67.09) - open/close 9671 (time 1776201348.09 total 150.11 last 66.13) total: 10000 open/close in 155.20 seconds: 64.43 ops/second - unlinked 0 (time 1776201357 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 698 (time 1776201461.11 total 10.00 last 69.80) - open/close 1322 (time 1776201471.12 total 20.01 last 62.33) - open/close 1987 (time 1776201481.14 total 30.03 last 66.35) - open/close 2634 (time 1776201491.14 total 40.04 last 64.68) - open/close 3300 (time 1776201501.14 total 50.04 last 66.59) - open/close 3964 (time 1776201511.15 total 60.05 last 66.33) - open/close 4642 (time 1776201521.18 total 70.07 last 67.66) - open/close 5305 (time 1776201531.19 total 80.09 last 66.18) - open/close 5990 (time 1776201541.20 total 90.09 last 68.46) - open/close 6626 (time 1776201551.21 total 100.10 last 63.56) - open/close 7270 (time 1776201561.21 total 110.10 last 64.39) - open/close 7952 (time 1776201571.23 total 120.12 last 68.08) - open/close 8593 (time 1776201581.23 total 130.13 last 64.05) - open/close 9261 (time 1776201591.24 total 140.14 last 66.74) - open/close 9952 (time 1776201601.25 total 150.14 last 69.08) total: 10000 open/close in 150.97 seconds: 66.24 ops/second - unlinked 0 (time 1776201607 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 618 (time 1776201717.28 total 10.01 last 61.73) - open/close 1177 (time 1776201727.30 total 20.03 last 55.77) - open/close 1774 (time 1776201737.32 total 30.05 last 59.63) - open/close 2264 (time 1776201747.33 total 40.06 last 48.95) - open/close 2808 (time 1776201757.33 total 50.06 last 54.39) - open/close 3392 (time 1776201767.34 total 60.07 last 58.33) - open/close 3980 (time 1776201777.36 total 70.09 last 58.70) - open/close 4512 (time 1776201787.37 total 80.10 last 53.13) - open/close 5106 (time 1776201797.37 total 90.10 last 59.39) - open/close 5623 (time 1776201807.38 total 100.11 last 51.65) - open/close 6210 (time 1776201817.38 total 110.11 last 58.67) - open/close 6817 (time 1776201827.40 total 120.13 last 60.63) - open/close 7426 (time 1776201837.41 total 130.14 last 60.83) - open/close 8001 (time 1776201847.42 total 140.14 last 57.45) - open/close 8606 (time 1776201857.42 total 150.15 last 60.47) - open/close 9193 (time 1776201867.43 total 160.16 last 58.64) - open/close 9752 (time 1776201877.44 total 170.17 last 55.85) total: 10000 open/close in 174.08 seconds: 57.44 ops/second - unlinked 0 (time 1776201886 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466339 s, 2.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg430-server Stopping /mnt/lustre-ost1 (opts:) on oleg430-server reboot facets: ost1 Failover ost1 to oleg430-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776202063.21 total 36.25 last 0.00) total: 100 open/close in 39.53 seconds: 2.53 ops/second umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg430-server: LNET ready to unload oleg430-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.