== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:36:47 (1761313007) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg407-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock oleg407-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 760 (time 1761313112.16 total 10.01 last 75.93) - open/close 1556 (time 1761313122.17 total 20.02 last 79.52) - open/close 2357 (time 1761313132.18 total 30.03 last 79.98) - open/close 3152 (time 1761313142.19 total 40.04 last 79.45) - open/close 3927 (time 1761313152.20 total 50.05 last 77.40) - open/close 4618 (time 1761313162.21 total 60.06 last 69.05) - open/close 5321 (time 1761313172.22 total 70.07 last 70.22) - open/close 6029 (time 1761313182.23 total 80.08 last 70.75) - open/close 6691 (time 1761313192.23 total 90.08 last 66.19) - open/close 7376 (time 1761313202.24 total 100.09 last 68.44) - open/close 8071 (time 1761313212.24 total 110.09 last 69.49) - open/close 8734 (time 1761313222.25 total 120.10 last 66.25) - open/close 9463 (time 1761313232.26 total 130.11 last 72.87) total: 10000 open/close in 136.25 seconds: 73.40 ops/second - unlinked 0 (time 1761313241 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 863 (time 1761313327.96 total 10.01 last 86.24) - open/close 1614 (time 1761313337.97 total 20.02 last 75.03) - open/close 2368 (time 1761313347.98 total 30.02 last 75.35) - open/close 3083 (time 1761313357.99 total 40.03 last 71.43) - open/close 3805 (time 1761313368.00 total 50.04 last 72.15) - open/close 4449 (time 1761313378.01 total 60.05 last 64.34) - open/close 5145 (time 1761313388.02 total 70.07 last 69.49) - open/close 5837 (time 1761313398.02 total 80.07 last 69.19) - open/close 6546 (time 1761313408.03 total 90.07 last 70.86) - open/close 7266 (time 1761313418.05 total 100.09 last 71.85) - open/close 7995 (time 1761313428.05 total 110.10 last 72.87) - open/close 8774 (time 1761313438.06 total 120.10 last 77.88) - open/close 9549 (time 1761313448.07 total 130.11 last 77.41) total: 10000 open/close in 136.18 seconds: 73.43 ops/second - unlinked 0 (time 1761313456 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 764 (time 1761313548.16 total 10.01 last 76.35) - open/close 1452 (time 1761313558.16 total 20.01 last 68.78) - open/close 2222 (time 1761313568.16 total 30.01 last 76.98) - open/close 3040 (time 1761313578.17 total 40.02 last 81.74) - open/close 3845 (time 1761313588.18 total 50.03 last 80.43) - open/close 4682 (time 1761313598.18 total 60.03 last 83.68) - open/close 5382 (time 1761313608.19 total 70.04 last 69.96) - open/close 6097 (time 1761313618.19 total 80.04 last 71.48) - open/close 6900 (time 1761313628.19 total 90.04 last 80.30) - open/close 7861 (time 1761313638.19 total 100.04 last 96.07) - open/close 8727 (time 1761313648.20 total 110.05 last 86.52) - open/close 9636 (time 1761313658.21 total 120.06 last 90.79) total: 10000 open/close in 124.43 seconds: 80.36 ops/second - unlinked 0 (time 1761313665 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 795 (time 1761313758.98 total 10.02 last 79.37) - open/close 1579 (time 1761313768.99 total 20.03 last 78.33) - open/close 2329 (time 1761313779.00 total 30.03 last 74.94) - open/close 3101 (time 1761313789.00 total 40.04 last 77.16) - open/close 3862 (time 1761313799.01 total 50.05 last 76.04) - open/close 4678 (time 1761313809.02 total 60.05 last 81.54) - open/close 5533 (time 1761313819.02 total 70.06 last 85.49) - open/close 6328 (time 1761313829.02 total 80.06 last 79.47) - open/close 7157 (time 1761313839.03 total 90.07 last 82.82) - open/close 7963 (time 1761313849.04 total 100.08 last 80.53) - open/close 8777 (time 1761313859.04 total 110.08 last 81.38) - open/close 9579 (time 1761313869.06 total 120.10 last 80.03) total: 10000 open/close in 125.84 seconds: 79.46 ops/second - unlinked 0 (time 1761313878 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 649 (time 1761313964.89 total 10.00 last 64.90) - open/close 1383 (time 1761313974.90 total 20.01 last 73.35) - open/close 2146 (time 1761313984.91 total 30.02 last 76.17) - open/close 2887 (time 1761313994.92 total 40.03 last 74.06) - open/close 3610 (time 1761314004.93 total 50.04 last 72.19) - open/close 4364 (time 1761314014.95 total 60.05 last 75.32) - open/close 5096 (time 1761314024.95 total 70.06 last 73.14) - open/close 5849 (time 1761314034.96 total 80.07 last 75.25) - open/close 6645 (time 1761314044.96 total 90.07 last 79.59) - open/close 7397 (time 1761314054.96 total 100.07 last 75.20) - open/close 8326 (time 1761314064.97 total 110.07 last 92.86) - open/close 9105 (time 1761314074.98 total 120.09 last 77.82) - open/close 9883 (time 1761314084.98 total 130.09 last 77.74) total: 10000 open/close in 131.68 seconds: 75.94 ops/second - unlinked 0 (time 1761314089 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 728 (time 1761314175.02 total 10.01 last 72.75) - open/close 1458 (time 1761314185.04 total 20.02 last 72.88) - open/close 2223 (time 1761314195.04 total 30.03 last 76.48) - open/close 3015 (time 1761314205.04 total 40.03 last 79.18) - open/close 3832 (time 1761314215.04 total 50.03 last 81.70) - open/close 4676 (time 1761314225.05 total 60.03 last 84.36) - open/close 5511 (time 1761314235.05 total 70.04 last 83.48) - open/close 6286 (time 1761314245.05 total 80.04 last 77.48) - open/close 7046 (time 1761314255.05 total 90.04 last 76.00) - open/close 7801 (time 1761314265.06 total 100.05 last 75.43) - open/close 8596 (time 1761314275.07 total 110.05 last 79.45) - open/close 9563 (time 1761314285.08 total 120.06 last 96.61) total: 10000 open/close in 124.95 seconds: 80.03 ops/second - unlinked 0 (time 1761314292 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 793 (time 1761314371.87 total 10.01 last 79.19) - open/close 1759 (time 1761314381.87 total 20.01 last 96.59) - open/close 2568 (time 1761314391.88 total 30.02 last 80.84) - open/close 3355 (time 1761314401.89 total 40.03 last 78.61) - open/close 4209 (time 1761314411.89 total 50.04 last 85.37) - open/close 5053 (time 1761314421.90 total 60.04 last 84.34) - open/close 5879 (time 1761314431.91 total 70.05 last 82.56) - open/close 6713 (time 1761314441.91 total 80.05 last 83.40) - open/close 7491 (time 1761314451.91 total 90.06 last 77.73) - open/close 8316 (time 1761314461.92 total 100.07 last 82.44) - open/close 9065 (time 1761314471.93 total 110.07 last 74.87) - open/close 9874 (time 1761314481.93 total 120.07 last 80.87) total: 10000 open/close in 121.56 seconds: 82.26 ops/second - unlinked 0 (time 1761314486 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 702 (time 1761314569.94 total 10.00 last 70.19) - open/close 1428 (time 1761314579.95 total 20.01 last 72.54) - open/close 2189 (time 1761314589.95 total 30.01 last 76.07) - open/close 2964 (time 1761314599.96 total 40.02 last 77.45) - open/close 3654 (time 1761314609.96 total 50.03 last 68.97) - open/close 4393 (time 1761314619.96 total 60.03 last 73.90) - open/close 5099 (time 1761314629.97 total 70.04 last 70.52) - open/close 5846 (time 1761314639.99 total 80.05 last 74.58) - open/close 6629 (time 1761314650.00 total 90.06 last 78.24) - open/close 7400 (time 1761314660.00 total 100.06 last 77.08) - open/close 8117 (time 1761314670.00 total 110.06 last 71.69) - open/close 8969 (time 1761314680.00 total 120.07 last 85.17) - open/close 9883 (time 1761314690.01 total 130.07 last 91.36) total: 10000 open/close in 131.59 seconds: 75.99 ops/second - unlinked 0 (time 1761314694 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 773 (time 1761314786.03 total 10.00 last 77.29) - open/close 1565 (time 1761314796.03 total 20.00 last 79.20) - open/close 2334 (time 1761314806.04 total 30.01 last 76.87) - open/close 3037 (time 1761314816.05 total 40.02 last 70.23) - open/close 3769 (time 1761314826.06 total 50.03 last 73.10) - open/close 4466 (time 1761314836.07 total 60.04 last 69.64) - open/close 5216 (time 1761314846.09 total 70.06 last 74.87) - open/close 5998 (time 1761314856.09 total 80.06 last 78.16) - open/close 6825 (time 1761314866.10 total 90.07 last 82.66) - open/close 7600 (time 1761314876.10 total 100.07 last 77.48) - open/close 8442 (time 1761314886.10 total 110.07 last 84.20) - open/close 9243 (time 1761314896.11 total 120.08 last 80.02) total: 10000 open/close in 129.66 seconds: 77.12 ops/second - unlinked 0 (time 1761314909 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 679 (time 1761314999.55 total 10.01 last 67.84) - open/close 1336 (time 1761315009.55 total 20.01 last 65.67) - open/close 2044 (time 1761315019.56 total 30.02 last 70.75) - open/close 2745 (time 1761315029.56 total 40.02 last 70.09) - open/close 3376 (time 1761315039.56 total 50.03 last 63.07) - open/close 4078 (time 1761315049.57 total 60.03 last 70.15) - open/close 4773 (time 1761315059.58 total 70.05 last 69.41) - open/close 5455 (time 1761315069.58 total 80.05 last 68.20) - open/close 6181 (time 1761315079.59 total 90.05 last 72.57) - open/close 6862 (time 1761315089.60 total 100.06 last 68.04) - open/close 7592 (time 1761315099.60 total 110.06 last 72.98) - open/close 8359 (time 1761315109.60 total 120.07 last 76.66) - open/close 9133 (time 1761315119.62 total 130.08 last 77.30) - open/close 9950 (time 1761315129.63 total 140.09 last 81.61) total: 10000 open/close in 140.84 seconds: 71.00 ops/second - unlinked 0 (time 1761315134 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 743 (time 1761315229.29 total 10.00 last 74.27) - open/close 1486 (time 1761315239.30 total 20.01 last 74.26) - open/close 2232 (time 1761315249.31 total 30.02 last 74.53) - open/close 2993 (time 1761315259.33 total 40.04 last 75.96) - open/close 3695 (time 1761315269.33 total 50.04 last 70.19) - open/close 4401 (time 1761315279.33 total 60.04 last 70.59) - open/close 5088 (time 1761315289.35 total 70.06 last 68.59) - open/close 5843 (time 1761315299.36 total 80.07 last 75.43) - open/close 6579 (time 1761315309.36 total 90.07 last 73.59) - open/close 7292 (time 1761315319.36 total 100.07 last 71.28) - open/close 7972 (time 1761315329.37 total 110.08 last 67.94) - open/close 8647 (time 1761315339.37 total 120.08 last 67.47) - open/close 9345 (time 1761315349.38 total 130.09 last 69.77) total: 10000 open/close in 139.13 seconds: 71.88 ops/second - unlinked 0 (time 1761315362 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44176 s, 2.4 MB/s fail_loc=0x80000204 Failing ost1 on oleg407-server Stopping /mnt/lustre-ost1 (opts:) on oleg407-server reboot facets: ost1 Failover ost1 to oleg407-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761315532.68 total 36.23 last 0.00) total: 100 open/close in 38.41 seconds: 2.60 ops/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg407-server: LNET ready to unload oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.