-----============= acceptance-small: conf-sanity ============----- Tue Oct 14 10:25:05 EDT 2025 oleg431-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg431-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg431-server oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg431-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg431-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg431-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg431-server: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg431-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:26:38 (1760451998) Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg431-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg431-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock oleg431-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1910 (time 1760452052.64 total 10.00 last 190.99) - open/close 3612 (time 1760452062.64 total 20.00 last 170.16) - open/close 6056 (time 1760452072.64 total 30.01 last 244.31) - open/close 8611 (time 1760452082.65 total 40.01 last 255.38) - open/close 10000 (time 1760452087.66 total 45.02 last 277.37) - open/close 12387 (time 1760452097.66 total 55.02 last 238.65) - open/close 14323 (time 1760452107.66 total 65.02 last 193.54) - open/close 16231 (time 1760452117.66 total 75.02 last 190.79) - open/close 18496 (time 1760452127.66 total 85.02 last 226.49) - open/close 20000 (time 1760452134.20 total 91.56 last 230.16) - open/close 22360 (time 1760452144.20 total 101.56 last 235.98) - open/close 24486 (time 1760452154.20 total 111.57 last 212.49) - open/close 26318 (time 1760452164.21 total 121.57 last 183.13) - open/close 27978 (time 1760452174.21 total 131.58 last 165.88) - open/close 29688 (time 1760452184.22 total 141.58 last 170.92) - open/close 31829 (time 1760452194.22 total 151.58 last 214.03) - open/close 33906 (time 1760452204.22 total 161.59 last 207.65) - open/close 36444 (time 1760452214.22 total 171.59 last 253.79) - open/close 38630 (time 1760452224.23 total 181.59 last 218.53) - open/close 40000 (time 1760452230.24 total 187.60 last 227.97) - open/close 41748 (time 1760452240.24 total 197.61 last 174.67) - open/close 43438 (time 1760452250.25 total 207.61 last 168.89) - open/close 45131 (time 1760452260.25 total 217.61 last 169.28) - open/close 47013 (time 1760452270.26 total 227.62 last 188.08) - open/close 49051 (time 1760452280.26 total 237.62 last 203.79) - open/close 50000 (time 1760452284.51 total 241.87 last 223.43) - open/close 52058 (time 1760452294.51 total 251.87 last 205.69) - open/close 53567 (time 1760452304.52 total 261.88 last 150.85) - open/close 55183 (time 1760452314.52 total 271.88 last 161.52) - open/close 57078 (time 1760452324.53 total 281.89 last 189.36) - open/close 58804 (time 1760452334.53 total 291.89 last 172.56) - open/close 60000 (time 1760452340.91 total 298.28 last 187.36) - open/close 61889 (time 1760452350.92 total 308.28 last 188.80) - open/close 64015 (time 1760452360.92 total 318.29 last 212.51) total: 64768 open/close in 323.12 seconds: 200.45 ops/second - /unlink 2972 (time 1760452377.61 total 10.00 last 297.15) - /unlink 6422 (time 1760452387.61 total 20.00 last 344.98) - /unlink 9881 (time 1760452397.61 total 30.00 last 345.86) - /unlink 13566 (time 1760452407.61 total 40.01 last 368.41) - /unlink 17086 (time 1760452417.62 total 50.01 last 351.85) - /unlink 20000 (time 1760452426.34 total 58.73 last 334.06) - /unlink 23623 (time 1760452436.34 total 68.73 last 362.26) - /unlink 27039 (time 1760452446.34 total 78.74 last 341.59) - /unlink 30000 (time 1760452454.65 total 87.04 last 356.51) - /unlink 33709 (time 1760452464.65 total 97.04 last 370.86) - /unlink 37309 (time 1760452474.65 total 107.04 last 359.93) - /unlink 40000 (time 1760452482.54 total 114.93 last 341.08) - /unlink 43497 (time 1760452492.54 total 124.94 last 349.58) - /unlink 46698 (time 1760452502.54 total 134.94 last 320.09) - /unlink 50000 (time 1760452512.00 total 144.40 last 349.04) - /unlink 54230 (time 1760452522.00 total 154.40 last 422.97) - /unlink 57914 (time 1760452532.01 total 164.40 last 368.34) - /unlink 60000 (time 1760452538.04 total 170.43 last 345.95) - /unlink 64717 (time 1760452548.04 total 180.43 last 471.55) total: 64768 /unlink in 180.62 seconds: 358.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2296 (time 1760452576.24 total 10.00 last 229.53) - open/close 4213 (time 1760452586.25 total 20.01 last 191.60) - open/close 6539 (time 1760452596.25 total 30.01 last 232.52) - open/close 8888 (time 1760452606.26 total 40.01 last 234.83) - open/close 10000 (time 1760452610.51 total 44.27 last 261.43) - open/close 12465 (time 1760452620.51 total 54.27 last 246.45) - open/close 14606 (time 1760452630.51 total 64.27 last 214.06) - open/close 16947 (time 1760452640.51 total 74.27 last 234.09) - open/close 19426 (time 1760452650.51 total 84.27 last 247.89) - open/close 20000 (time 1760452652.83 total 86.59 last 248.25) - open/close 22248 (time 1760452662.83 total 96.59 last 224.76) - open/close 24331 (time 1760452672.83 total 106.59 last 208.30) - open/close 26548 (time 1760452682.83 total 116.59 last 221.66) - open/close 28755 (time 1760452692.83 total 126.59 last 220.61) - open/close 30000 (time 1760452699.94 total 133.70 last 175.10) - open/close 31893 (time 1760452709.95 total 143.71 last 189.22) - open/close 33920 (time 1760452719.95 total 153.71 last 202.66) - open/close 35922 (time 1760452729.95 total 163.71 last 200.12) - open/close 37997 (time 1760452739.96 total 173.72 last 207.41) - open/close 40000 (time 1760452747.68 total 181.44 last 259.27) - open/close 42763 (time 1760452757.68 total 191.44 last 276.27) - open/close 45464 (time 1760452767.69 total 201.45 last 270.08) - open/close 47913 (time 1760452777.69 total 211.45 last 244.84) - open/close 50000 (time 1760452786.36 total 220.12 last 240.64) - open/close 52210 (time 1760452796.36 total 230.12 last 220.97) - open/close 54338 (time 1760452806.36 total 240.12 last 212.78) - open/close 56875 (time 1760452816.36 total 250.12 last 253.70) - open/close 59181 (time 1760452826.37 total 260.13 last 230.51) - open/close 60000 (time 1760452829.80 total 263.56 last 238.29) - open/close 62571 (time 1760452839.80 total 273.56 last 257.08) total: 64768 open/close in 281.14 seconds: 230.38 ops/second - /unlink 5821 (time 1760452858.89 total 10.00 last 582.06) - /unlink 10000 (time 1760452866.85 total 17.96 last 525.00) - /unlink 14789 (time 1760452876.85 total 27.96 last 478.88) - /unlink 20000 (time 1760452884.72 total 35.83 last 662.11) - /unlink 26063 (time 1760452894.72 total 45.83 last 606.28) - /unlink 30000 (time 1760452900.85 total 51.96 last 641.99) - /unlink 35385 (time 1760452910.85 total 61.96 last 538.49) - /unlink 40000 (time 1760452919.12 total 70.23 last 558.29) - /unlink 45878 (time 1760452929.12 total 80.23 last 587.79) - /unlink 50000 (time 1760452936.27 total 87.38 last 576.72) - /unlink 56769 (time 1760452946.28 total 97.40 last 675.71) - /unlink 60000 (time 1760452950.95 total 102.06 last 692.37) total: 64768 /unlink in 109.35 seconds: 592.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4074 (time 1760452985.36 total 10.00 last 407.32) - open/close 8745 (time 1760452995.36 total 20.00 last 467.04) - open/close 10000 (time 1760452997.98 total 22.62 last 479.19) - open/close 14777 (time 1760453007.98 total 32.62 last 477.66) - open/close 19145 (time 1760453017.98 total 42.62 last 436.77) - open/close 23996 (time 1760453027.98 total 52.62 last 485.07) - open/close 28454 (time 1760453037.98 total 62.63 last 445.74) - open/close 30000 (time 1760453041.22 total 65.87 last 477.09) - open/close 35094 (time 1760453051.22 total 75.87 last 509.39) - open/close 39667 (time 1760453061.22 total 85.87 last 457.22) - open/close 44572 (time 1760453071.22 total 95.87 last 490.50) - open/close 49136 (time 1760453081.22 total 105.87 last 456.40) - open/close 53573 (time 1760453091.22 total 115.87 last 443.66) - open/close 58143 (time 1760453101.23 total 125.87 last 456.96) - open/close 60000 (time 1760453105.68 total 130.33 last 416.71) - open/close 64612 (time 1760453115.78 total 140.42 last 456.81) total: 64768 open/close in 140.75 seconds: 460.17 ops/second - /unlink 7387 (time 1760453127.41 total 10.00 last 738.63) - /unlink 10000 (time 1760453130.31 total 12.89 last 903.27) - /unlink 18152 (time 1760453140.31 total 22.89 last 815.19) - /unlink 20000 (time 1760453143.18 total 25.77 last 643.37) - /unlink 27480 (time 1760453153.18 total 35.77 last 747.97) - /unlink 30000 (time 1760453155.76 total 38.35 last 976.00) - /unlink 37798 (time 1760453165.76 total 48.35 last 779.70) - /unlink 40000 (time 1760453169.36 total 51.95 last 612.22) - /unlink 47780 (time 1760453179.36 total 61.95 last 777.96) - /unlink 50000 (time 1760453181.72 total 64.30 last 941.94) - /unlink 58177 (time 1760453191.72 total 74.30 last 817.64) - /unlink 60000 (time 1760453194.13 total 76.72 last 755.32) total: 64768 /unlink in 82.05 seconds: 789.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4771 (time 1760453231.87 total 10.00 last 477.05) - open/close 9971 (time 1760453241.87 total 20.00 last 519.90) - open/close 14510 (time 1760453251.87 total 30.00 last 453.81) - open/close 19579 (time 1760453261.87 total 40.01 last 506.85) - open/close 23903 (time 1760453271.88 total 50.01 last 432.28) - open/close 28424 (time 1760453281.88 total 60.01 last 452.07) - open/close 30000 (time 1760453285.33 total 63.46 last 457.13) - open/close 34538 (time 1760453295.33 total 73.46 last 453.64) - open/close 38601 (time 1760453305.33 total 83.46 last 406.25) - open/close 40000 (time 1760453308.94 total 87.07 last 387.92) - open/close 44125 (time 1760453318.94 total 97.07 last 412.46) - open/close 48588 (time 1760453328.94 total 107.07 last 446.26) - open/close 50000 (time 1760453332.10 total 110.23 last 446.34) - open/close 54743 (time 1760453342.10 total 120.23 last 474.27) - open/close 59900 (time 1760453352.10 total 130.23 last 515.66) total: 64768 open/close in 140.07 seconds: 462.39 ops/second - /unlink 7613 (time 1760453373.26 total 10.00 last 761.20) - /unlink 10000 (time 1760453376.16 total 12.90 last 822.06) - /unlink 17212 (time 1760453386.16 total 22.91 last 721.16) - /unlink 20000 (time 1760453390.22 total 26.96 last 687.94) - /unlink 27927 (time 1760453400.22 total 36.96 last 792.69) - /unlink 30000 (time 1760453403.26 total 40.01 last 680.23) - /unlink 38342 (time 1760453413.27 total 50.01 last 834.09) - /unlink 40000 (time 1760453415.73 total 52.47 last 672.71) - /unlink 48153 (time 1760453425.73 total 62.47 last 815.26) - /unlink 50000 (time 1760453427.82 total 64.56 last 886.13) - /unlink 58489 (time 1760453437.82 total 74.56 last 848.38) - /unlink 60000 (time 1760453439.99 total 76.73 last 697.31) total: 64768 /unlink in 82.18 seconds: 788.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4999 (time 1760453477.74 total 10.00 last 499.88) - open/close 9916 (time 1760453487.75 total 20.00 last 491.67) - open/close 14383 (time 1760453497.75 total 30.00 last 446.70) - open/close 19071 (time 1760453507.75 total 40.00 last 468.74) - open/close 23550 (time 1760453517.75 total 50.00 last 447.88) - open/close 28804 (time 1760453527.75 total 60.00 last 525.35) - open/close 30000 (time 1760453530.05 total 62.30 last 520.01) - open/close 34997 (time 1760453540.05 total 72.30 last 499.69) - open/close 39886 (time 1760453550.05 total 82.30 last 488.88) - open/close 44651 (time 1760453560.05 total 92.30 last 476.47) - open/close 49765 (time 1760453570.05 total 102.31 last 511.33) - open/close 54172 (time 1760453580.05 total 112.31 last 440.69) - open/close 58400 (time 1760453590.05 total 122.31 last 422.76) - open/close 60000 (time 1760453593.98 total 126.24 last 407.10) - open/close 64113 (time 1760453603.98 total 136.24 last 411.30) total: 64768 open/close in 137.57 seconds: 470.80 ops/second - /unlink 8358 (time 1760453616.69 total 10.00 last 835.76) - /unlink 10000 (time 1760453619.17 total 12.49 last 660.72) - /unlink 17473 (time 1760453629.18 total 22.49 last 747.13) - /unlink 20000 (time 1760453632.63 total 25.95 last 730.65) - /unlink 27134 (time 1760453642.64 total 35.95 last 713.11) - /unlink 30000 (time 1760453646.27 total 39.58 last 790.31) - /unlink 38271 (time 1760453656.27 total 49.58 last 827.09) - /unlink 45495 (time 1760453666.27 total 59.58 last 722.34) - /unlink 50000 (time 1760453672.33 total 65.64 last 743.38) - /unlink 57125 (time 1760453682.33 total 75.64 last 712.43) - /unlink 60000 (time 1760453686.63 total 79.94 last 668.39) total: 64768 /unlink in 86.30 seconds: 750.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg431-server: LNET ready to unload oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 PASS 106 (1732s) Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1849 sec ========== 10:55:55 (1760453755)