== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:18:36 (1781281116) Stopping clients: oleg627-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg627-client.virtnet /mnt/lustre2 (opts:-f) oleg627-server: oleg627-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg627-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg627-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg627-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock oleg627-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 793 (time 1781281269.81 total 10.01 last 79.26) - open/close 1611 (time 1781281279.81 total 20.01 last 81.74) - open/close 2418 (time 1781281289.82 total 30.02 last 80.67) - open/close 3258 (time 1781281299.82 total 40.02 last 83.99) - open/close 4073 (time 1781281309.83 total 50.03 last 81.38) - open/close 4889 (time 1781281319.85 total 60.04 last 81.50) - open/close 5677 (time 1781281329.85 total 70.05 last 78.78) - open/close 6488 (time 1781281339.86 total 80.06 last 81.00) - open/close 7271 (time 1781281349.87 total 90.07 last 78.23) - open/close 8090 (time 1781281359.87 total 100.07 last 81.88) - open/close 8879 (time 1781281369.88 total 110.08 last 78.86) - open/close 9692 (time 1781281379.88 total 120.08 last 81.28) total: 10000 open/close in 123.63 seconds: 80.89 ops/second - unlinked 0 (time 1781281387 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 790 (time 1781281474.12 total 10.01 last 78.93) - open/close 1597 (time 1781281484.13 total 20.02 last 80.64) - open/close 2438 (time 1781281494.13 total 30.02 last 84.09) - open/close 3265 (time 1781281504.15 total 40.04 last 82.51) - open/close 4105 (time 1781281514.17 total 50.05 last 83.90) - open/close 4915 (time 1781281524.17 total 60.05 last 80.99) - open/close 5794 (time 1781281534.17 total 70.06 last 87.85) - open/close 6692 (time 1781281544.17 total 80.06 last 89.80) - open/close 7542 (time 1781281554.18 total 90.06 last 84.97) - open/close 8426 (time 1781281564.18 total 100.07 last 88.36) - open/close 9345 (time 1781281574.19 total 110.07 last 91.85) total: 10000 open/close in 117.29 seconds: 85.26 ops/second - unlinked 0 (time 1781281584 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 821 (time 1781281668.65 total 10.01 last 82.03) - open/close 1649 (time 1781281678.66 total 20.02 last 82.68) - open/close 2463 (time 1781281688.67 total 30.03 last 81.38) - open/close 3333 (time 1781281698.68 total 40.04 last 86.87) - open/close 4186 (time 1781281708.69 total 50.05 last 85.23) - open/close 5019 (time 1781281718.70 total 60.05 last 83.26) - open/close 5925 (time 1781281728.70 total 70.06 last 90.53) - open/close 6776 (time 1781281738.71 total 80.07 last 85.02) - open/close 7620 (time 1781281748.72 total 90.08 last 84.36) - open/close 8517 (time 1781281758.72 total 100.08 last 89.67) - open/close 9369 (time 1781281768.73 total 110.09 last 85.13) total: 10000 open/close in 117.97 seconds: 84.76 ops/second - unlinked 0 (time 1781281780 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 833 (time 1781281858.77 total 10.00 last 83.29) - open/close 1753 (time 1781281868.79 total 20.01 last 91.88) - open/close 2636 (time 1781281878.79 total 30.02 last 88.28) - open/close 3537 (time 1781281888.79 total 40.02 last 90.05) - open/close 4482 (time 1781281898.81 total 50.04 last 94.34) - open/close 5527 (time 1781281908.82 total 60.04 last 104.44) - open/close 6612 (time 1781281918.82 total 70.05 last 108.46) - open/close 7530 (time 1781281928.84 total 80.07 last 91.64) - open/close 8502 (time 1781281938.84 total 90.07 last 97.16) - open/close 9399 (time 1781281948.86 total 100.08 last 89.58) total: 10000 open/close in 106.86 seconds: 93.58 ops/second - unlinked 0 (time 1781281958 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 936 (time 1781282031.93 total 10.01 last 93.53) - open/close 1842 (time 1781282041.94 total 20.02 last 90.50) - open/close 2880 (time 1781282051.95 total 30.03 last 103.71) - open/close 3780 (time 1781282061.96 total 40.04 last 89.90) - open/close 4784 (time 1781282071.96 total 50.04 last 100.40) - open/close 5706 (time 1781282081.97 total 60.04 last 92.19) - open/close 6625 (time 1781282091.97 total 70.04 last 91.85) - open/close 7573 (time 1781282101.97 total 80.05 last 94.77) - open/close 8637 (time 1781282111.98 total 90.05 last 106.35) - open/close 9711 (time 1781282121.98 total 100.05 last 107.38) total: 10000 open/close in 102.95 seconds: 97.14 ops/second - unlinked 0 (time 1781282128 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 867 (time 1781282206.93 total 10.02 last 86.56) - open/close 1864 (time 1781282216.94 total 20.02 last 99.62) - open/close 2783 (time 1781282226.94 total 30.03 last 91.88) - open/close 3729 (time 1781282236.95 total 40.03 last 94.53) - open/close 4620 (time 1781282246.95 total 50.04 last 89.08) - open/close 5561 (time 1781282256.97 total 60.05 last 93.96) - open/close 6511 (time 1781282266.97 total 70.06 last 94.95) - open/close 7374 (time 1781282276.98 total 80.07 last 86.22) - open/close 8365 (time 1781282286.98 total 90.07 last 99.09) - open/close 9229 (time 1781282296.99 total 100.07 last 86.34) total: 10000 open/close in 109.03 seconds: 91.72 ops/second - unlinked 0 (time 1781282309 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 975 (time 1781282389.75 total 10.00 last 97.46) - open/close 1922 (time 1781282399.75 total 20.01 last 94.67) - open/close 2898 (time 1781282409.75 total 30.01 last 97.56) - open/close 3850 (time 1781282419.75 total 40.01 last 95.19) - open/close 4761 (time 1781282429.76 total 50.02 last 91.03) - open/close 5637 (time 1781282439.76 total 60.02 last 87.60) - open/close 6550 (time 1781282449.77 total 70.02 last 91.26) - open/close 7420 (time 1781282459.77 total 80.03 last 86.92) - open/close 8329 (time 1781282469.78 total 90.04 last 90.87) - open/close 9191 (time 1781282479.78 total 100.04 last 86.16) total: 10000 open/close in 109.58 seconds: 91.26 ops/second - unlinked 0 (time 1781282493 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 814 (time 1781282575.95 total 10.01 last 81.33) - open/close 1662 (time 1781282585.96 total 20.02 last 84.70) - open/close 2487 (time 1781282595.96 total 30.02 last 82.50) - open/close 3353 (time 1781282605.97 total 40.03 last 86.51) - open/close 4306 (time 1781282615.97 total 50.04 last 95.26) - open/close 5257 (time 1781282625.98 total 60.04 last 95.05) - open/close 6152 (time 1781282635.98 total 70.05 last 89.46) - open/close 7013 (time 1781282645.99 total 80.05 last 86.08) - open/close 7820 (time 1781282655.99 total 90.05 last 80.67) - open/close 8660 (time 1781282666.00 total 100.06 last 83.94) - open/close 9517 (time 1781282676.00 total 110.06 last 85.64) total: 10000 open/close in 115.76 seconds: 86.39 ops/second - unlinked 0 (time 1781282685 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 823 (time 1781282769.36 total 10.00 last 82.29) - open/close 1766 (time 1781282779.36 total 20.00 last 94.29) - open/close 2673 (time 1781282789.37 total 30.01 last 90.62) - open/close 3636 (time 1781282799.38 total 40.02 last 96.26) - open/close 4541 (time 1781282809.38 total 50.02 last 90.47) - open/close 5407 (time 1781282819.38 total 60.02 last 86.58) - open/close 6233 (time 1781282829.39 total 70.03 last 82.49) - open/close 7032 (time 1781282839.40 total 80.04 last 79.83) - open/close 7823 (time 1781282849.41 total 90.05 last 79.04) - open/close 8682 (time 1781282859.41 total 100.05 last 85.89) - open/close 9567 (time 1781282869.42 total 110.06 last 88.46) total: 10000 open/close in 115.34 seconds: 86.70 ops/second - unlinked 0 (time 1781282878 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 899 (time 1781282961.95 total 10.01 last 89.83) - open/close 1849 (time 1781282971.96 total 20.01 last 94.94) - open/close 2755 (time 1781282981.97 total 30.02 last 90.50) - open/close 3687 (time 1781282991.97 total 40.03 last 93.18) - open/close 4652 (time 1781283001.97 total 50.03 last 96.45) - open/close 5508 (time 1781283011.98 total 60.03 last 85.59) - open/close 6415 (time 1781283021.98 total 70.03 last 90.68) - open/close 7283 (time 1781283031.98 total 80.04 last 86.78) - open/close 8139 (time 1781283041.99 total 90.04 last 85.54) - open/close 9008 (time 1781283051.99 total 100.05 last 86.89) - open/close 9867 (time 1781283061.99 total 110.05 last 85.89) total: 10000 open/close in 111.65 seconds: 89.56 ops/second - unlinked 0 (time 1781283067 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 889 (time 1781283142.16 total 10.02 last 88.72) - open/close 1973 (time 1781283152.17 total 20.03 last 108.32) - open/close 2941 (time 1781283162.17 total 30.03 last 96.79) - open/close 3846 (time 1781283172.18 total 40.04 last 90.43) - open/close 4667 (time 1781283182.18 total 50.04 last 82.10) - open/close 5601 (time 1781283192.18 total 60.04 last 93.36) - open/close 6372 (time 1781283202.19 total 70.04 last 77.07) - open/close 7167 (time 1781283212.19 total 80.05 last 79.47) - open/close 8017 (time 1781283222.20 total 90.06 last 84.93) - open/close 8876 (time 1781283232.20 total 100.06 last 85.85) - open/close 9714 (time 1781283242.22 total 110.08 last 83.66) total: 10000 open/close in 113.64 seconds: 88.00 ops/second - unlinked 0 (time 1781283249 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5376 376192 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371196 s, 2.8 MB/s fail_loc=0x80000204 Failing ost1 on oleg627-server Stopping /mnt/lustre-ost1 (opts:) on oleg627-server reboot facets: ost1 Failover ost1 to oleg627-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.16 seconds: 46.27 ops/second umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg627-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg627-server: LNET ready to unload oleg627-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.