-----============= acceptance-small: conf-sanity ============----- Tue Jun 3 18:16:07 EDT 2025 oleg302-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg302-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg302-server oleg302-server: oleg302-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg302-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg302-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg302-server: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg302-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:17:04 (1748989024) Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) oleg302-server: oleg302-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg302-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg302-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock oleg302-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5652 (time 1748989057.97 total 10.00 last 565.19) - open/close 10000 (time 1748989066.01 total 18.04 last 540.72) - open/close 15450 (time 1748989076.01 total 28.04 last 544.98) - open/close 20000 (time 1748989084.25 total 36.27 last 552.77) - open/close 25620 (time 1748989094.25 total 46.27 last 561.94) - open/close 30000 (time 1748989102.03 total 54.06 last 562.78) - open/close 35476 (time 1748989112.03 total 64.06 last 547.53) - open/close 40000 (time 1748989120.17 total 72.20 last 555.87) - open/close 43846 (time 1748989130.17 total 82.20 last 384.48) - open/close 47606 (time 1748989140.17 total 92.20 last 375.96) - open/close 50000 (time 1748989146.10 total 98.12 last 404.25) - open/close 54004 (time 1748989156.10 total 108.12 last 400.37) - open/close 57820 (time 1748989166.10 total 118.13 last 381.54) - open/close 60000 (time 1748989170.57 total 122.59 last 487.92) - open/close 64435 (time 1748989180.57 total 132.59 last 443.47) total: 64768 open/close in 133.29 seconds: 485.93 ops/second - /unlink 9082 (time 1748989192.20 total 10.00 last 908.15) - /unlink 17450 (time 1748989202.20 total 20.00 last 836.73) - /unlink 20000 (time 1748989205.23 total 23.04 last 840.31) - /unlink 27716 (time 1748989215.23 total 33.04 last 771.56) - /unlink 30000 (time 1748989218.56 total 36.37 last 685.13) - /unlink 38265 (time 1748989228.56 total 46.37 last 826.50) - /unlink 40000 (time 1748989230.73 total 48.54 last 800.65) - /unlink 47569 (time 1748989240.73 total 58.54 last 756.83) - /unlink 50000 (time 1748989243.57 total 61.38 last 855.91) - /unlink 57090 (time 1748989253.57 total 71.38 last 708.98) - /unlink 60000 (time 1748989256.73 total 74.53 last 922.32) total: 64768 /unlink in 80.53 seconds: 804.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2562 (time 1748989293.54 total 10.00 last 256.17) - open/close 5333 (time 1748989303.54 total 20.00 last 277.05) - open/close 9008 (time 1748989313.54 total 30.00 last 367.49) - open/close 10000 (time 1748989316.12 total 32.58 last 385.24) - open/close 12873 (time 1748989326.12 total 42.58 last 287.28) - open/close 15469 (time 1748989336.12 total 52.58 last 259.60) - open/close 18705 (time 1748989346.12 total 62.58 last 323.58) - open/close 20000 (time 1748989348.93 total 65.40 last 459.80) - open/close 23854 (time 1748989358.93 total 75.40 last 385.39) - open/close 26639 (time 1748989368.94 total 85.40 last 278.42) - open/close 29971 (time 1748989378.94 total 95.40 last 333.17) - open/close 32514 (time 1748989388.94 total 105.40 last 254.26) - open/close 35500 (time 1748989398.94 total 115.40 last 298.53) - open/close 38055 (time 1748989408.94 total 125.41 last 255.46) - open/close 40000 (time 1748989416.94 total 133.40 last 243.33) - open/close 42703 (time 1748989426.94 total 143.40 last 270.23) - open/close 46039 (time 1748989436.94 total 153.40 last 333.57) - open/close 49549 (time 1748989446.94 total 163.40 last 350.96) - open/close 53895 (time 1748989456.94 total 173.40 last 434.58) - open/close 58067 (time 1748989466.94 total 183.41 last 417.16) - open/close 60000 (time 1748989470.91 total 187.37 last 487.19) - open/close 63667 (time 1748989480.91 total 197.37 last 366.69) total: 64768 open/close in 199.75 seconds: 324.25 ops/second - /unlink 9009 (time 1748989494.10 total 10.00 last 900.86) - /unlink 18434 (time 1748989504.10 total 20.00 last 942.40) - /unlink 26757 (time 1748989514.10 total 30.00 last 832.29) - /unlink 30000 (time 1748989517.71 total 33.61 last 899.90) - /unlink 38873 (time 1748989527.71 total 43.61 last 887.30) - /unlink 47976 (time 1748989537.71 total 53.61 last 910.30) - /unlink 50000 (time 1748989539.90 total 55.80 last 922.14) - /unlink 56980 (time 1748989549.90 total 65.80 last 697.92) - /unlink 60000 (time 1748989553.57 total 69.47 last 824.12) total: 64768 /unlink in 76.91 seconds: 842.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4052 (time 1748989594.19 total 10.00 last 405.13) - open/close 8576 (time 1748989604.19 total 20.00 last 452.34) - open/close 10000 (time 1748989608.65 total 24.46 last 319.55) - open/close 13978 (time 1748989618.65 total 34.46 last 397.65) - open/close 16856 (time 1748989628.66 total 44.46 last 287.77) - open/close 20000 (time 1748989638.05 total 53.86 last 334.59) - open/close 23871 (time 1748989648.05 total 63.86 last 387.06) - open/close 27215 (time 1748989658.06 total 73.86 last 334.29) - open/close 29978 (time 1748989668.06 total 83.87 last 276.24) - open/close 32455 (time 1748989678.06 total 93.87 last 247.66) - open/close 35004 (time 1748989688.06 total 103.87 last 254.81) - open/close 37608 (time 1748989698.06 total 113.87 last 260.39) - open/close 40000 (time 1748989707.52 total 123.33 last 252.95) - open/close 42510 (time 1748989717.52 total 133.33 last 250.91) - open/close 45138 (time 1748989727.53 total 143.34 last 262.73) - open/close 47580 (time 1748989737.53 total 153.34 last 244.14) - open/close 50000 (time 1748989746.89 total 162.70 last 258.40) - open/close 52544 (time 1748989756.89 total 172.70 last 254.40) - open/close 55111 (time 1748989766.90 total 182.70 last 256.67) - open/close 57634 (time 1748989776.97 total 192.78 last 250.51) - open/close 60000 (time 1748989786.04 total 201.85 last 260.73) - open/close 62589 (time 1748989796.04 total 211.85 last 258.86) total: 64768 open/close in 220.63 seconds: 293.56 ops/second - /unlink 4727 (time 1748989816.11 total 10.00 last 472.65) - /unlink 9422 (time 1748989826.11 total 20.00 last 469.42) - /unlink 14155 (time 1748989836.11 total 30.00 last 473.22) - /unlink 18984 (time 1748989846.11 total 40.01 last 482.86) - /unlink 20000 (time 1748989848.45 total 42.34 last 435.63) - /unlink 24722 (time 1748989858.45 total 52.34 last 472.19) - /unlink 29812 (time 1748989868.45 total 62.34 last 508.97) - /unlink 34598 (time 1748989878.45 total 72.34 last 478.46) - /unlink 39392 (time 1748989888.45 total 82.34 last 479.28) - /unlink 44127 (time 1748989898.45 total 92.34 last 473.48) - /unlink 49229 (time 1748989908.45 total 102.34 last 510.17) - /unlink 53996 (time 1748989918.46 total 112.35 last 476.48) - /unlink 60000 (time 1748989928.26 total 122.15 last 612.38) total: 64768 /unlink in 131.72 seconds: 491.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5014 (time 1748989970.11 total 10.00 last 501.40) - open/close 9333 (time 1748989980.12 total 20.00 last 431.79) - open/close 12579 (time 1748989990.12 total 30.00 last 324.56) - open/close 15027 (time 1748990000.12 total 40.01 last 244.77) - open/close 17469 (time 1748990010.12 total 50.01 last 244.18) - open/close 19899 (time 1748990020.12 total 60.01 last 242.93) - open/close 22390 (time 1748990030.12 total 70.01 last 249.07) - open/close 24908 (time 1748990040.12 total 80.01 last 251.77) - open/close 27355 (time 1748990050.13 total 90.01 last 244.64) - open/close 29776 (time 1748990060.13 total 100.01 last 242.08) - open/close 32217 (time 1748990070.13 total 110.02 last 244.08) - open/close 34772 (time 1748990080.13 total 120.02 last 255.49) - open/close 37267 (time 1748990090.13 total 130.02 last 249.49) - open/close 39767 (time 1748990100.13 total 140.02 last 249.95) - open/close 42587 (time 1748990110.13 total 150.02 last 281.95) - open/close 45807 (time 1748990120.13 total 160.02 last 321.99) - open/close 50000 (time 1748990129.66 total 169.55 last 440.22) - open/close 54223 (time 1748990139.66 total 179.55 last 422.17) - open/close 58605 (time 1748990149.66 total 189.55 last 438.18) - open/close 60000 (time 1748990153.86 total 193.74 last 332.47) - open/close 63709 (time 1748990163.86 total 203.75 last 370.83) total: 64768 open/close in 206.42 seconds: 313.76 ops/second - /unlink 5817 (time 1748990177.51 total 10.00 last 581.65) - /unlink 10000 (time 1748990184.34 total 16.83 last 612.79) - /unlink 14912 (time 1748990194.34 total 26.83 last 491.10) - /unlink 19989 (time 1748990204.34 total 36.83 last 507.66) - /unlink 24745 (time 1748990214.34 total 46.83 last 475.57) - /unlink 29323 (time 1748990224.34 total 56.83 last 457.80) - /unlink 33908 (time 1748990234.34 total 66.83 last 458.42) - /unlink 38572 (time 1748990244.34 total 76.83 last 466.36) - /unlink 40000 (time 1748990247.32 total 79.81 last 480.15) - /unlink 44502 (time 1748990257.32 total 89.81 last 450.18) - /unlink 49088 (time 1748990267.32 total 99.81 last 458.53) - /unlink 50000 (time 1748990269.38 total 101.87 last 442.53) - /unlink 56995 (time 1748990279.38 total 111.87 last 699.42) - /unlink 60000 (time 1748990284.77 total 117.26 last 557.38) total: 64768 /unlink in 127.16 seconds: 509.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3332 (time 1748990325.15 total 10.00 last 333.07) - open/close 6686 (time 1748990335.15 total 20.00 last 335.38) - open/close 9557 (time 1748990345.15 total 30.01 last 287.03) - open/close 12360 (time 1748990355.16 total 40.01 last 280.21) - open/close 15539 (time 1748990365.16 total 50.01 last 317.81) - open/close 18322 (time 1748990375.16 total 60.02 last 278.21) - open/close 20000 (time 1748990381.26 total 66.12 last 275.08) - open/close 22577 (time 1748990391.27 total 76.12 last 257.62) - open/close 25074 (time 1748990401.27 total 86.12 last 249.62) - open/close 27651 (time 1748990411.27 total 96.12 last 257.63) - open/close 30000 (time 1748990418.03 total 102.88 last 347.80) - open/close 33474 (time 1748990428.03 total 112.88 last 347.35) - open/close 36421 (time 1748990438.03 total 122.88 last 294.67) - open/close 40000 (time 1748990447.95 total 132.80 last 360.87) - open/close 42897 (time 1748990457.95 total 142.80 last 289.69) - open/close 46316 (time 1748990467.95 total 152.80 last 341.87) - open/close 48736 (time 1748990477.95 total 162.80 last 241.93) - open/close 50000 (time 1748990483.07 total 167.92 last 246.88) - open/close 52435 (time 1748990493.07 total 177.93 last 243.40) - open/close 55435 (time 1748990503.08 total 187.93 last 299.95) - open/close 58482 (time 1748990513.08 total 197.93 last 304.64) - open/close 60000 (time 1748990518.25 total 203.10 last 293.68) - open/close 62869 (time 1748990528.25 total 213.10 last 286.85) total: 64768 open/close in 221.07 seconds: 292.97 ops/second - /unlink 4493 (time 1748990547.43 total 10.00 last 449.28) - /unlink 9098 (time 1748990557.43 total 20.00 last 460.48) - /unlink 10000 (time 1748990559.43 total 22.00 last 450.79) - /unlink 14747 (time 1748990569.43 total 32.00 last 474.69) - /unlink 19469 (time 1748990579.43 total 42.00 last 472.17) - /unlink 24253 (time 1748990589.43 total 52.00 last 478.39) - /unlink 28983 (time 1748990599.43 total 62.00 last 472.94) - /unlink 33730 (time 1748990609.43 total 72.01 last 474.63) - /unlink 38396 (time 1748990619.44 total 82.01 last 466.53) - /unlink 40000 (time 1748990622.16 total 84.73 last 589.48) - /unlink 44719 (time 1748990632.16 total 94.73 last 471.85) - /unlink 49731 (time 1748990642.16 total 104.73 last 501.12) - /unlink 54286 (time 1748990652.16 total 114.73 last 455.48) - /unlink 59034 (time 1748990662.16 total 124.73 last 474.78) - /unlink 60000 (time 1748990664.27 total 126.84 last 458.23) total: 64768 /unlink in 136.48 seconds: 474.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server LNET ready to unload unloading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: LNET ready to unload modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 PASS 106 (1690s) Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 oleg302-server: oleg302-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1770 sec ========== 18:45:37 (1748990737)