== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 10:31:50 (1761316310) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) oleg123-server: oleg123-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg123-server: detected 4 online CPUs by sysfs oleg123-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg123-server: detected 4 online CPUs by sysfs oleg123-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg123-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 962 (time 1761316391.11 total 10.00 last 96.17) - open/close 1903 (time 1761316401.11 total 20.00 last 94.08) - open/close 2853 (time 1761316411.12 total 30.01 last 94.93) - open/close 3739 (time 1761316421.13 total 40.02 last 88.52) - open/close 4622 (time 1761316431.15 total 50.04 last 88.12) - open/close 5470 (time 1761316441.15 total 60.04 last 84.77) - open/close 6344 (time 1761316451.15 total 70.04 last 87.39) - open/close 7164 (time 1761316461.16 total 80.05 last 81.95) - open/close 8016 (time 1761316471.16 total 90.06 last 85.16) - open/close 8852 (time 1761316481.17 total 100.06 last 83.53) - open/close 9671 (time 1761316491.17 total 110.07 last 81.88) total: 10000 open/close in 114.21 seconds: 87.56 ops/second - unlinked 0 (time 1761316498 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 852 (time 1761316579.03 total 10.01 last 85.13) - open/close 1728 (time 1761316589.04 total 20.01 last 87.56) - open/close 2561 (time 1761316599.05 total 30.02 last 83.21) - open/close 3364 (time 1761316609.05 total 40.03 last 80.27) - open/close 4383 (time 1761316619.06 total 50.03 last 101.88) - open/close 5249 (time 1761316629.06 total 60.04 last 86.53) - open/close 6143 (time 1761316639.07 total 70.05 last 89.30) - open/close 7066 (time 1761316649.08 total 80.05 last 92.27) - open/close 8047 (time 1761316659.08 total 90.05 last 98.09) - open/close 8944 (time 1761316669.08 total 100.05 last 89.70) - open/close 9987 (time 1761316679.08 total 110.06 last 104.28) total: 10000 open/close in 110.19 seconds: 90.75 ops/second - unlinked 0 (time 1761316681 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 864 (time 1761316748.11 total 10.01 last 86.34) - open/close 1639 (time 1761316758.12 total 20.01 last 77.48) - open/close 2445 (time 1761316768.12 total 30.02 last 80.56) - open/close 3309 (time 1761316778.13 total 40.02 last 86.37) - open/close 4205 (time 1761316788.13 total 50.02 last 89.59) - open/close 5208 (time 1761316798.13 total 60.03 last 100.24) - open/close 6172 (time 1761316808.13 total 70.03 last 96.38) - open/close 7065 (time 1761316818.14 total 80.03 last 89.26) - open/close 8001 (time 1761316828.15 total 90.04 last 93.48) - open/close 8977 (time 1761316838.15 total 100.05 last 97.58) - open/close 9873 (time 1761316848.17 total 110.06 last 89.50) total: 10000 open/close in 111.49 seconds: 89.69 ops/second - unlinked 0 (time 1761316853 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 914 (time 1761316936.49 total 10.00 last 91.39) - open/close 1796 (time 1761316946.50 total 20.01 last 88.11) - open/close 2629 (time 1761316956.50 total 30.01 last 83.30) - open/close 3472 (time 1761316966.51 total 40.02 last 84.22) - open/close 4285 (time 1761316976.52 total 50.03 last 81.23) - open/close 5133 (time 1761316986.52 total 60.03 last 84.80) - open/close 5984 (time 1761316996.53 total 70.04 last 84.99) - open/close 6857 (time 1761317006.54 total 80.05 last 87.22) - open/close 7697 (time 1761317016.54 total 90.06 last 83.96) - open/close 8553 (time 1761317026.55 total 100.06 last 85.55) - open/close 9338 (time 1761317036.56 total 110.07 last 78.45) total: 10000 open/close in 117.62 seconds: 85.02 ops/second - unlinked 0 (time 1761317047 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 840 (time 1761317132.01 total 10.01 last 83.94) - open/close 1755 (time 1761317142.02 total 20.03 last 91.33) - open/close 2588 (time 1761317152.03 total 30.03 last 83.25) - open/close 3348 (time 1761317162.04 total 40.04 last 75.96) - open/close 4156 (time 1761317172.04 total 50.04 last 80.76) - open/close 4999 (time 1761317182.05 total 60.05 last 84.22) - open/close 5772 (time 1761317192.05 total 70.05 last 77.30) - open/close 6561 (time 1761317202.06 total 80.06 last 78.85) - open/close 7353 (time 1761317212.06 total 90.06 last 79.17) - open/close 8168 (time 1761317222.09 total 100.09 last 81.28) - open/close 8985 (time 1761317232.10 total 110.10 last 81.62) - open/close 9767 (time 1761317242.11 total 120.11 last 78.16) total: 10000 open/close in 123.02 seconds: 81.29 ops/second - unlinked 0 (time 1761317248 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 814 (time 1761317332.17 total 10.01 last 81.36) - open/close 1631 (time 1761317342.17 total 20.01 last 81.67) - open/close 2482 (time 1761317352.17 total 30.01 last 85.09) - open/close 3343 (time 1761317362.18 total 40.02 last 86.06) - open/close 4174 (time 1761317372.19 total 50.02 last 83.04) - open/close 5009 (time 1761317382.20 total 60.03 last 83.42) - open/close 5956 (time 1761317392.20 total 70.04 last 94.67) - open/close 6709 (time 1761317402.20 total 80.04 last 75.26) - open/close 7474 (time 1761317412.21 total 90.05 last 76.46) - open/close 8238 (time 1761317422.22 total 100.06 last 76.30) - open/close 9055 (time 1761317432.24 total 110.07 last 81.59) - open/close 9897 (time 1761317442.24 total 120.07 last 84.19) total: 10000 open/close in 121.19 seconds: 82.52 ops/second - unlinked 0 (time 1761317446 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 778 (time 1761317533.29 total 10.01 last 77.76) - open/close 1530 (time 1761317543.30 total 20.01 last 75.15) - open/close 2335 (time 1761317553.30 total 30.02 last 80.47) - open/close 3211 (time 1761317563.31 total 40.02 last 87.53) - open/close 4019 (time 1761317573.32 total 50.04 last 80.68) - open/close 4807 (time 1761317583.33 total 60.04 last 78.76) - open/close 5649 (time 1761317593.33 total 70.04 last 84.19) - open/close 6532 (time 1761317603.34 total 80.05 last 88.24) - open/close 7377 (time 1761317613.34 total 90.06 last 84.46) - open/close 8244 (time 1761317623.35 total 100.06 last 86.66) - open/close 9160 (time 1761317633.36 total 110.07 last 91.51) total: 10000 open/close in 117.94 seconds: 84.79 ops/second - unlinked 0 (time 1761317643 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 814 (time 1761317720.98 total 10.01 last 81.30) - open/close 1624 (time 1761317730.98 total 20.01 last 80.99) - open/close 2565 (time 1761317740.99 total 30.02 last 94.06) - open/close 3492 (time 1761317750.99 total 40.02 last 92.69) - open/close 4517 (time 1761317761.00 total 50.03 last 102.34) - open/close 5461 (time 1761317771.00 total 60.04 last 94.39) - open/close 6288 (time 1761317781.01 total 70.04 last 82.68) - open/close 7125 (time 1761317791.01 total 80.04 last 83.67) - open/close 7945 (time 1761317801.01 total 90.04 last 82.00) - open/close 8782 (time 1761317811.01 total 100.04 last 83.67) - open/close 9662 (time 1761317821.02 total 110.05 last 87.97) total: 10000 open/close in 113.25 seconds: 88.30 ops/second - unlinked 0 (time 1761317826 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 904 (time 1761317894.20 total 10.00 last 90.38) - open/close 1779 (time 1761317904.20 total 20.00 last 87.48) - open/close 2723 (time 1761317914.21 total 30.01 last 94.31) - open/close 3671 (time 1761317924.22 total 40.02 last 94.70) - open/close 4578 (time 1761317934.23 total 50.03 last 90.63) - open/close 5535 (time 1761317944.24 total 60.04 last 95.62) - open/close 6407 (time 1761317954.24 total 70.04 last 87.19) - open/close 7258 (time 1761317964.24 total 80.05 last 85.06) - open/close 8133 (time 1761317974.25 total 90.06 last 87.40) - open/close 8934 (time 1761317984.26 total 100.06 last 80.05) - open/close 9717 (time 1761317994.26 total 110.07 last 78.27) total: 10000 open/close in 113.59 seconds: 88.03 ops/second - unlinked 0 (time 1761318001 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 819 (time 1761318079.38 total 10.01 last 81.82) - open/close 1726 (time 1761318089.38 total 20.01 last 90.70) - open/close 2590 (time 1761318099.38 total 30.01 last 86.37) - open/close 3457 (time 1761318109.39 total 40.02 last 86.64) - open/close 4283 (time 1761318119.40 total 50.03 last 82.55) - open/close 5123 (time 1761318129.40 total 60.03 last 83.97) - open/close 6038 (time 1761318139.41 total 70.04 last 91.42) - open/close 7024 (time 1761318149.41 total 80.04 last 98.58) - open/close 8036 (time 1761318159.42 total 90.05 last 101.12) - open/close 8998 (time 1761318169.43 total 100.06 last 96.12) total: 10000 open/close in 109.45 seconds: 91.37 ops/second - unlinked 0 (time 1761318181 ; total 1 ; last 1) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 812 (time 1761318260.88 total 10.00 last 81.18) - open/close 1646 (time 1761318270.88 total 20.00 last 83.38) - open/close 2481 (time 1761318280.89 total 30.02 last 83.39) - open/close 3345 (time 1761318290.90 total 40.02 last 86.38) - open/close 4233 (time 1761318300.91 total 50.04 last 88.65) - open/close 5104 (time 1761318310.92 total 60.04 last 87.05) - open/close 6023 (time 1761318320.92 total 70.04 last 91.88) - open/close 7015 (time 1761318330.93 total 80.05 last 99.12) - open/close 7936 (time 1761318340.93 total 90.05 last 92.10) - open/close 8799 (time 1761318350.94 total 100.06 last 86.25) - open/close 9668 (time 1761318360.94 total 110.06 last 86.88) total: 10000 open/close in 113.70 seconds: 87.95 ops/second - unlinked 0 (time 1761318367 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229773 s, 4.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg123-server Stopping /mnt/lustre-ost1 (opts:) on oleg123-server reboot facets: ost1 Failover ost1 to oleg123-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761318511.67 total 39.61 last 0.00) total: 100 open/close in 41.69 seconds: 2.40 ops/second umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg123-server: LNET ready to unload oleg123-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.