== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:48:12 (1761313692) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) oleg442-server: oleg442-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg442-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 762 (time 1761313798.23 total 10.02 last 76.05) - open/close 1558 (time 1761313808.23 total 20.02 last 79.59) - open/close 2400 (time 1761313818.24 total 30.03 last 84.10) - open/close 3181 (time 1761313828.24 total 40.04 last 78.06) - open/close 3985 (time 1761313838.26 total 50.05 last 80.29) - open/close 4773 (time 1761313848.26 total 60.05 last 78.78) - open/close 5566 (time 1761313858.26 total 70.06 last 79.27) - open/close 6369 (time 1761313868.27 total 80.07 last 80.22) - open/close 7098 (time 1761313878.28 total 90.07 last 72.88) - open/close 7864 (time 1761313888.28 total 100.08 last 76.55) - open/close 8612 (time 1761313898.29 total 110.09 last 74.73) - open/close 9412 (time 1761313908.30 total 120.10 last 79.94) total: 10000 open/close in 127.57 seconds: 78.39 ops/second - unlinked 0 (time 1761313918 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 736 (time 1761314012.69 total 10.02 last 73.48) - open/close 1510 (time 1761314022.71 total 20.04 last 77.24) - open/close 2296 (time 1761314032.71 total 30.04 last 78.58) - open/close 3088 (time 1761314042.72 total 40.05 last 79.13) - open/close 3867 (time 1761314052.73 total 50.06 last 77.82) - open/close 4760 (time 1761314062.74 total 60.07 last 89.22) - open/close 5518 (time 1761314072.74 total 70.07 last 75.79) - open/close 6303 (time 1761314082.75 total 80.07 last 78.48) - open/close 7078 (time 1761314092.75 total 90.07 last 77.47) - open/close 7828 (time 1761314102.76 total 100.08 last 74.95) - open/close 8623 (time 1761314112.76 total 110.08 last 79.49) - open/close 9330 (time 1761314122.77 total 120.09 last 70.65) total: 10000 open/close in 129.29 seconds: 77.35 ops/second - unlinked 0 (time 1761314135 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 836 (time 1761314223.23 total 10.00 last 83.58) - open/close 1605 (time 1761314233.24 total 20.02 last 76.80) - open/close 2387 (time 1761314243.24 total 30.02 last 78.16) - open/close 3215 (time 1761314253.25 total 40.02 last 82.78) - open/close 3939 (time 1761314263.25 total 50.03 last 72.34) - open/close 4729 (time 1761314273.26 total 60.04 last 78.93) - open/close 5672 (time 1761314283.27 total 70.05 last 94.22) - open/close 6520 (time 1761314293.29 total 80.07 last 84.66) - open/close 7258 (time 1761314303.29 total 90.07 last 73.76) - open/close 8033 (time 1761314313.33 total 100.11 last 77.23) - open/close 8867 (time 1761314323.33 total 110.11 last 83.38) - open/close 9705 (time 1761314333.33 total 120.11 last 83.80) total: 10000 open/close in 123.72 seconds: 80.83 ops/second - unlinked 0 (time 1761314339 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 822 (time 1761314421.19 total 10.00 last 82.16) - open/close 1636 (time 1761314431.19 total 20.01 last 81.39) - open/close 2438 (time 1761314441.19 total 30.01 last 80.20) - open/close 3231 (time 1761314451.19 total 40.01 last 79.29) - open/close 4058 (time 1761314461.21 total 50.03 last 82.51) - open/close 4795 (time 1761314471.22 total 60.03 last 73.68) - open/close 5605 (time 1761314481.22 total 70.04 last 80.94) - open/close 6417 (time 1761314491.23 total 80.05 last 81.15) - open/close 7243 (time 1761314501.23 total 90.05 last 82.59) - open/close 8052 (time 1761314511.24 total 100.05 last 80.85) - open/close 8796 (time 1761314521.24 total 110.05 last 74.40) - open/close 9574 (time 1761314531.24 total 120.06 last 77.78) total: 10000 open/close in 125.50 seconds: 79.68 ops/second - unlinked 0 (time 1761314539 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 768 (time 1761314635.25 total 10.01 last 76.72) - open/close 1543 (time 1761314645.25 total 20.01 last 77.48) - open/close 2320 (time 1761314655.25 total 30.01 last 77.69) - open/close 3106 (time 1761314665.26 total 40.02 last 78.53) - open/close 3878 (time 1761314675.27 total 50.03 last 77.17) - open/close 4757 (time 1761314685.27 total 60.03 last 87.87) - open/close 5591 (time 1761314695.27 total 70.03 last 83.37) - open/close 6287 (time 1761314705.28 total 80.04 last 69.54) - open/close 7005 (time 1761314715.29 total 90.05 last 71.72) - open/close 7708 (time 1761314725.30 total 100.06 last 70.27) - open/close 8397 (time 1761314735.30 total 110.06 last 68.87) - open/close 9079 (time 1761314745.31 total 120.07 last 68.12) - open/close 9797 (time 1761314755.31 total 130.08 last 71.78) total: 10000 open/close in 132.87 seconds: 75.26 ops/second - unlinked 0 (time 1761314761 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 730 (time 1761314852.03 total 10.00 last 73.00) - open/close 1488 (time 1761314862.03 total 20.00 last 75.77) - open/close 2267 (time 1761314872.03 total 30.01 last 77.88) - open/close 3073 (time 1761314882.04 total 40.02 last 80.53) - open/close 3894 (time 1761314892.06 total 50.03 last 81.94) - open/close 4676 (time 1761314902.07 total 60.04 last 78.16) - open/close 5364 (time 1761314912.07 total 70.04 last 68.77) - open/close 6081 (time 1761314922.10 total 80.08 last 71.47) - open/close 6718 (time 1761314932.10 total 90.08 last 63.69) - open/close 7364 (time 1761314942.11 total 100.09 last 64.53) - open/close 8066 (time 1761314952.12 total 110.10 last 70.14) - open/close 8825 (time 1761314962.13 total 120.11 last 75.82) - open/close 9622 (time 1761314972.14 total 130.12 last 79.61) total: 10000 open/close in 135.55 seconds: 73.77 ops/second - unlinked 0 (time 1761314981 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 747 (time 1761315078.00 total 10.00 last 74.67) - open/close 1426 (time 1761315088.02 total 20.02 last 67.77) - open/close 2210 (time 1761315098.02 total 30.03 last 78.38) - open/close 3004 (time 1761315108.03 total 40.03 last 79.35) - open/close 3799 (time 1761315118.04 total 50.04 last 79.42) - open/close 4592 (time 1761315128.04 total 60.04 last 79.29) - open/close 5331 (time 1761315138.04 total 70.04 last 73.89) - open/close 6016 (time 1761315148.06 total 80.06 last 68.37) - open/close 6638 (time 1761315158.06 total 90.06 last 62.19) - open/close 7388 (time 1761315168.08 total 100.08 last 74.88) - open/close 8127 (time 1761315178.09 total 110.09 last 73.83) - open/close 8848 (time 1761315188.09 total 120.09 last 72.08) - open/close 9584 (time 1761315198.11 total 130.11 last 73.49) total: 10000 open/close in 135.72 seconds: 73.68 ops/second - unlinked 0 (time 1761315207 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 762 (time 1761315298.79 total 10.01 last 76.14) - open/close 1530 (time 1761315308.80 total 20.02 last 76.73) - open/close 2223 (time 1761315318.82 total 30.03 last 69.19) - open/close 2912 (time 1761315328.82 total 40.04 last 68.87) - open/close 3620 (time 1761315338.82 total 50.04 last 70.78) - open/close 4326 (time 1761315348.83 total 60.05 last 70.56) - open/close 5066 (time 1761315358.84 total 70.05 last 73.95) - open/close 5721 (time 1761315368.84 total 80.06 last 65.46) - open/close 6421 (time 1761315378.86 total 90.07 last 69.90) - open/close 7095 (time 1761315388.86 total 100.08 last 67.38) - open/close 7795 (time 1761315398.87 total 110.09 last 69.91) - open/close 8504 (time 1761315408.88 total 120.10 last 70.84) - open/close 9199 (time 1761315418.89 total 130.10 last 69.47) - open/close 9866 (time 1761315428.90 total 140.12 last 66.60) total: 10000 open/close in 142.13 seconds: 70.36 ops/second - unlinked 0 (time 1761315435 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 662 (time 1761315537.17 total 10.01 last 66.11) - open/close 1312 (time 1761315547.18 total 20.02 last 64.94) - open/close 2011 (time 1761315557.19 total 30.03 last 69.86) - open/close 2725 (time 1761315567.19 total 40.03 last 71.39) - open/close 3357 (time 1761315577.21 total 50.05 last 63.10) - open/close 4018 (time 1761315587.21 total 60.05 last 66.10) - open/close 4637 (time 1761315597.21 total 70.05 last 61.90) - open/close 5274 (time 1761315607.21 total 80.05 last 63.67) - open/close 5982 (time 1761315617.23 total 90.07 last 70.67) - open/close 6683 (time 1761315627.24 total 100.08 last 70.05) - open/close 7343 (time 1761315637.24 total 110.08 last 65.97) - open/close 7988 (time 1761315647.25 total 120.09 last 64.44) - open/close 8690 (time 1761315657.28 total 130.12 last 69.97) - open/close 9350 (time 1761315667.29 total 140.13 last 65.93) total: 10000 open/close in 149.80 seconds: 66.76 ops/second - unlinked 0 (time 1761315681 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 645 (time 1761315785.72 total 10.01 last 64.46) - open/close 1326 (time 1761315795.72 total 20.01 last 68.06) - open/close 2052 (time 1761315805.73 total 30.02 last 72.57) - open/close 2760 (time 1761315815.73 total 40.02 last 70.77) - open/close 3465 (time 1761315825.74 total 50.03 last 70.46) - open/close 4121 (time 1761315835.75 total 60.04 last 65.54) - open/close 4768 (time 1761315845.75 total 70.04 last 64.69) - open/close 5493 (time 1761315855.76 total 80.05 last 72.44) - open/close 6174 (time 1761315865.77 total 90.06 last 68.02) - open/close 6899 (time 1761315875.78 total 100.07 last 72.40) - open/close 7519 (time 1761315885.81 total 110.10 last 61.85) - open/close 8128 (time 1761315895.83 total 120.12 last 60.74) - open/close 8651 (time 1761315905.84 total 130.13 last 52.25) - open/close 9247 (time 1761315915.85 total 140.14 last 59.54) - open/close 9934 (time 1761315925.86 total 150.15 last 68.63) total: 10000 open/close in 151.15 seconds: 66.16 ops/second - unlinked 0 (time 1761315931 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 663 (time 1761316030.05 total 10.00 last 66.28) - open/close 1339 (time 1761316040.06 total 20.01 last 67.58) - open/close 2103 (time 1761316050.06 total 30.01 last 76.33) - open/close 2842 (time 1761316060.09 total 40.04 last 73.74) - open/close 3540 (time 1761316070.09 total 50.04 last 69.79) - open/close 4312 (time 1761316080.09 total 60.04 last 77.15) - open/close 5043 (time 1761316090.10 total 70.05 last 73.04) - open/close 5749 (time 1761316100.11 total 80.06 last 70.56) - open/close 6423 (time 1761316110.12 total 90.07 last 67.35) - open/close 7146 (time 1761316120.13 total 100.08 last 72.19) - open/close 7856 (time 1761316130.14 total 110.09 last 70.94) - open/close 8620 (time 1761316140.15 total 120.10 last 76.31) - open/close 9307 (time 1761316150.16 total 130.11 last 68.68) - open/close 9951 (time 1761316160.17 total 140.12 last 64.31) total: 10000 open/close in 140.90 seconds: 70.97 ops/second - unlinked 0 (time 1761316164 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320608 s, 3.3 MB/s fail_loc=0x80000204 Failing ost1 on oleg442-server Stopping /mnt/lustre-ost1 (opts:) on oleg442-server reboot facets: ost1 Failover ost1 to oleg442-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761316324.51 total 36.60 last 0.00) total: 100 open/close in 38.87 seconds: 2.57 ops/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg442-server: LNET ready to unload oleg442-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.