== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 13:13:34 (1761239614) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) oleg449-server: oleg449-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg449-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 972 (time 1761239726.56 total 10.01 last 97.13) - open/close 1926 (time 1761239736.56 total 20.01 last 95.39) - open/close 2832 (time 1761239746.57 total 30.01 last 90.54) - open/close 3704 (time 1761239756.57 total 40.02 last 87.16) - open/close 4626 (time 1761239766.59 total 50.03 last 92.08) - open/close 5620 (time 1761239776.59 total 60.04 last 99.35) - open/close 6556 (time 1761239786.59 total 70.04 last 93.59) - open/close 7542 (time 1761239796.60 total 80.05 last 98.52) - open/close 8511 (time 1761239806.60 total 90.05 last 96.89) - open/close 9458 (time 1761239816.61 total 100.06 last 94.63) total: 10000 open/close in 105.33 seconds: 94.94 ops/second - unlinked 0 (time 1761239824 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1306 (time 1761239888.42 total 10.00 last 130.58) - open/close 2528 (time 1761239898.42 total 20.00 last 122.17) - open/close 3832 (time 1761239908.43 total 30.01 last 130.35) - open/close 5147 (time 1761239918.43 total 40.02 last 131.39) - open/close 6650 (time 1761239928.44 total 50.02 last 150.29) - open/close 7659 (time 1761239938.44 total 60.02 last 100.89) - open/close 8615 (time 1761239948.44 total 70.02 last 95.59) - open/close 9509 (time 1761239958.44 total 80.02 last 89.36) total: 10000 open/close in 86.20 seconds: 116.01 ops/second - unlinked 0 (time 1761239968 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 749 (time 1761240036.88 total 10.01 last 74.84) - open/close 1543 (time 1761240046.89 total 20.01 last 79.35) - open/close 2289 (time 1761240056.89 total 30.01 last 74.60) - open/close 3053 (time 1761240066.90 total 40.02 last 76.32) - open/close 3914 (time 1761240076.91 total 50.03 last 86.02) - open/close 4708 (time 1761240086.91 total 60.04 last 79.38) - open/close 5609 (time 1761240096.92 total 70.04 last 90.02) - open/close 6390 (time 1761240106.92 total 80.04 last 78.10) - open/close 7153 (time 1761240116.92 total 90.05 last 76.30) - open/close 7879 (time 1761240126.92 total 100.05 last 72.60) - open/close 8608 (time 1761240136.93 total 110.06 last 72.81) - open/close 9337 (time 1761240146.93 total 120.06 last 72.89) total: 10000 open/close in 128.79 seconds: 77.65 ops/second - unlinked 0 (time 1761240158 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 783 (time 1761240235.80 total 10.00 last 78.27) - open/close 1596 (time 1761240245.81 total 20.01 last 81.23) - open/close 2347 (time 1761240255.83 total 30.03 last 74.97) - open/close 3116 (time 1761240265.83 total 40.03 last 76.86) - open/close 3857 (time 1761240275.85 total 50.05 last 73.99) - open/close 4689 (time 1761240285.86 total 60.06 last 83.11) - open/close 5476 (time 1761240295.86 total 70.06 last 78.69) - open/close 6293 (time 1761240305.86 total 80.06 last 81.68) - open/close 7085 (time 1761240315.87 total 90.07 last 79.18) - open/close 7899 (time 1761240325.88 total 100.08 last 81.30) - open/close 8761 (time 1761240335.88 total 110.08 last 86.19) - open/close 9599 (time 1761240345.88 total 120.08 last 83.77) total: 10000 open/close in 124.83 seconds: 80.11 ops/second - unlinked 0 (time 1761240353 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 736 (time 1761240428.39 total 10.01 last 73.56) - open/close 1478 (time 1761240438.39 total 20.01 last 74.15) - open/close 2214 (time 1761240448.42 total 30.03 last 73.44) - open/close 2985 (time 1761240458.43 total 40.05 last 77.00) - open/close 3732 (time 1761240468.43 total 50.05 last 74.68) - open/close 4414 (time 1761240478.44 total 60.06 last 68.15) - open/close 5143 (time 1761240488.44 total 70.06 last 72.88) - open/close 5845 (time 1761240498.45 total 80.07 last 70.15) - open/close 6563 (time 1761240508.47 total 90.09 last 71.66) - open/close 7359 (time 1761240518.47 total 100.09 last 79.57) - open/close 8139 (time 1761240528.48 total 110.10 last 77.95) - open/close 8842 (time 1761240538.50 total 120.12 last 70.14) - open/close 9561 (time 1761240548.50 total 130.12 last 71.89) total: 10000 open/close in 136.59 seconds: 73.21 ops/second - unlinked 0 (time 1761240558 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 560 (time 1761240650.84 total 10.00 last 55.97) - open/close 1361 (time 1761240660.84 total 20.01 last 80.06) - open/close 2148 (time 1761240670.85 total 30.02 last 78.66) - open/close 2984 (time 1761240680.86 total 40.02 last 83.52) - open/close 3724 (time 1761240690.86 total 50.03 last 73.99) - open/close 4581 (time 1761240700.86 total 60.03 last 85.68) - open/close 5397 (time 1761240710.86 total 70.03 last 81.58) - open/close 6141 (time 1761240720.87 total 80.04 last 74.34) - open/close 6964 (time 1761240730.88 total 90.05 last 82.22) - open/close 7778 (time 1761240740.89 total 100.06 last 81.31) - open/close 8613 (time 1761240750.90 total 110.07 last 83.44) - open/close 9469 (time 1761240760.91 total 120.07 last 85.54) total: 10000 open/close in 127.87 seconds: 78.20 ops/second - unlinked 0 (time 1761240772 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 779 (time 1761240859.33 total 10.01 last 77.83) - open/close 1618 (time 1761240869.33 total 20.02 last 83.83) - open/close 2453 (time 1761240879.34 total 30.02 last 83.47) - open/close 3238 (time 1761240889.34 total 40.02 last 78.49) - open/close 4091 (time 1761240899.36 total 50.04 last 85.14) - open/close 4938 (time 1761240909.37 total 60.05 last 84.64) - open/close 5808 (time 1761240919.37 total 70.05 last 86.99) - open/close 6522 (time 1761240929.37 total 80.05 last 71.39) - open/close 7289 (time 1761240939.37 total 90.05 last 76.68) - open/close 8163 (time 1761240949.38 total 100.06 last 87.32) - open/close 8966 (time 1761240959.39 total 110.07 last 80.24) - open/close 9798 (time 1761240969.40 total 120.08 last 83.13) total: 10000 open/close in 122.45 seconds: 81.67 ops/second - unlinked 0 (time 1761240975 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 869 (time 1761241057.57 total 10.01 last 86.82) - open/close 1737 (time 1761241067.57 total 20.01 last 86.79) - open/close 2656 (time 1761241077.59 total 30.03 last 91.75) - open/close 3488 (time 1761241087.59 total 40.03 last 83.19) - open/close 4315 (time 1761241097.59 total 50.03 last 82.67) - open/close 5171 (time 1761241107.60 total 60.04 last 85.53) - open/close 5997 (time 1761241117.61 total 70.05 last 82.53) - open/close 6851 (time 1761241127.61 total 80.05 last 85.40) - open/close 7713 (time 1761241137.61 total 90.05 last 86.19) - open/close 8611 (time 1761241147.61 total 100.05 last 89.79) - open/close 9399 (time 1761241157.62 total 110.06 last 78.76) total: 10000 open/close in 117.68 seconds: 84.98 ops/second - unlinked 0 (time 1761241168 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 856 (time 1761241251.07 total 10.01 last 85.55) - open/close 1727 (time 1761241261.08 total 20.01 last 87.06) - open/close 2560 (time 1761241271.08 total 30.02 last 83.23) - open/close 3263 (time 1761241281.09 total 40.03 last 70.25) - open/close 4058 (time 1761241291.09 total 50.03 last 79.48) - open/close 4963 (time 1761241301.10 total 60.03 last 90.49) - open/close 5903 (time 1761241311.10 total 70.03 last 93.96) - open/close 6811 (time 1761241321.11 total 80.04 last 90.75) - open/close 7692 (time 1761241331.11 total 90.05 last 88.02) - open/close 8630 (time 1761241341.12 total 100.06 last 93.72) - open/close 9531 (time 1761241351.12 total 110.06 last 90.10) total: 10000 open/close in 115.77 seconds: 86.38 ops/second - unlinked 0 (time 1761241360 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 947 (time 1761241433.25 total 10.01 last 94.65) - open/close 1889 (time 1761241443.27 total 20.02 last 94.06) - open/close 2741 (time 1761241453.27 total 30.02 last 85.19) - open/close 3567 (time 1761241463.27 total 40.02 last 82.57) - open/close 4516 (time 1761241473.28 total 50.03 last 94.83) - open/close 5425 (time 1761241483.28 total 60.03 last 90.88) - open/close 6378 (time 1761241493.29 total 70.04 last 95.22) - open/close 7227 (time 1761241503.30 total 80.05 last 84.83) - open/close 8092 (time 1761241513.31 total 90.06 last 86.39) - open/close 9011 (time 1761241523.32 total 100.07 last 91.82) - open/close 9963 (time 1761241533.32 total 110.08 last 95.14) total: 10000 open/close in 110.47 seconds: 90.52 ops/second - unlinked 0 (time 1761241536 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1086 (time 1761241610.75 total 10.01 last 108.47) - open/close 2016 (time 1761241620.76 total 20.01 last 92.98) - open/close 3008 (time 1761241630.76 total 30.02 last 99.19) - open/close 3902 (time 1761241640.77 total 40.02 last 89.33) - open/close 4833 (time 1761241650.77 total 50.03 last 93.06) - open/close 5749 (time 1761241660.78 total 60.03 last 91.53) - open/close 6733 (time 1761241670.78 total 70.04 last 98.40) - open/close 7753 (time 1761241680.78 total 80.04 last 101.95) - open/close 8712 (time 1761241690.78 total 90.04 last 95.89) - open/close 9721 (time 1761241700.79 total 100.05 last 100.85) total: 10000 open/close in 103.05 seconds: 97.04 ops/second - unlinked 0 (time 1761241706 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5120 376448 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225939 s, 4.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg449-server Stopping /mnt/lustre-ost1 (opts:) on oleg449-server reboot facets: ost1 Failover ost1 to oleg449-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761241843.11 total 33.49 last 0.00) total: 100 open/close in 35.22 seconds: 2.84 ops/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local oleg449-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg449-server: LNET ready to unload oleg449-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.