-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:01 EDT 2025 oleg136-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg136-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server oleg136-server: oleg136-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:46:45 (1747399605) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) oleg136-server: oleg136-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1544 (time 1747399663.24 total 10.00 last 154.39) - open/close 2921 (time 1747399673.25 total 20.01 last 137.59) - open/close 4484 (time 1747399683.25 total 30.01 last 156.27) - open/close 6035 (time 1747399693.26 total 40.02 last 155.01) - open/close 7394 (time 1747399703.26 total 50.02 last 135.84) - open/close 8603 (time 1747399713.27 total 60.03 last 120.82) - open/close 9794 (time 1747399723.27 total 70.03 last 119.09) - open/close 10000 (time 1747399725.29 total 72.04 last 102.26) - open/close 11035 (time 1747399735.29 total 82.05 last 103.44) - open/close 12095 (time 1747399745.29 total 92.05 last 105.98) - open/close 13161 (time 1747399755.30 total 102.06 last 106.55) - open/close 14280 (time 1747399765.31 total 112.06 last 111.81) - open/close 15508 (time 1747399775.31 total 122.07 last 122.75) - open/close 16765 (time 1747399785.32 total 132.07 last 125.64) - open/close 18053 (time 1747399795.32 total 142.07 last 128.80) - open/close 19339 (time 1747399805.32 total 152.07 last 128.57) - open/close 20000 (time 1747399810.41 total 157.17 last 129.67) - open/close 21243 (time 1747399820.42 total 167.18 last 124.26) - open/close 22652 (time 1747399830.42 total 177.18 last 140.83) - open/close 24267 (time 1747399840.43 total 187.18 last 161.46) - open/close 25742 (time 1747399850.43 total 197.19 last 147.44) - open/close 27183 (time 1747399860.43 total 207.19 last 144.04) - open/close 28619 (time 1747399870.44 total 217.20 last 143.53) - open/close 30000 (time 1747399880.10 total 226.86 last 142.95) - open/close 31481 (time 1747399890.10 total 236.86 last 148.07) - open/close 32837 (time 1747399900.10 total 246.86 last 135.57) - open/close 34593 (time 1747399910.11 total 256.86 last 175.56) - open/close 36519 (time 1747399920.11 total 266.86 last 192.59) - open/close 38408 (time 1747399930.11 total 276.87 last 188.85) - open/close 40000 (time 1747399939.42 total 286.18 last 170.96) - open/close 41844 (time 1747399949.43 total 296.18 last 184.31) - open/close 43609 (time 1747399959.43 total 306.19 last 176.46) - open/close 45463 (time 1747399969.43 total 316.19 last 185.38) - open/close 47242 (time 1747399979.43 total 326.19 last 177.85) - open/close 48717 (time 1747399989.44 total 336.19 last 147.44) - open/close 50000 (time 1747399997.53 total 344.28 last 158.58) - open/close 51695 (time 1747400007.53 total 354.28 last 169.50) - open/close 53589 (time 1747400017.53 total 364.29 last 189.33) - open/close 55756 (time 1747400027.53 total 374.29 last 216.64) - open/close 57661 (time 1747400037.54 total 384.29 last 190.46) - open/close 59407 (time 1747400047.54 total 394.30 last 174.52) - open/close 60000 (time 1747400050.42 total 397.18 last 205.98) - open/close 61840 (time 1747400060.42 total 407.18 last 183.94) - open/close 63964 (time 1747400070.42 total 417.18 last 212.38) total: 64768 open/close in 420.87 seconds: 153.89 ops/second - /unlink 3969 (time 1747400085.43 total 10.00 last 396.84) - /unlink 7643 (time 1747400095.43 total 20.00 last 367.34) - /unlink 10000 (time 1747400102.09 total 26.67 last 353.60) - /unlink 13823 (time 1747400112.10 total 36.67 last 382.18) - /unlink 18135 (time 1747400122.10 total 46.67 last 431.15) - /unlink 20000 (time 1747400125.63 total 50.21 last 527.26) - /unlink 24313 (time 1747400135.64 total 60.21 last 431.23) - /unlink 28883 (time 1747400145.64 total 70.21 last 456.94) - /unlink 30000 (time 1747400147.68 total 72.25 last 548.08) - /unlink 34902 (time 1747400157.68 total 82.25 last 490.17) - /unlink 40000 (time 1747400166.30 total 90.88 last 591.04) - /unlink 45115 (time 1747400176.30 total 100.88 last 511.49) - /unlink 50000 (time 1747400185.65 total 110.23 last 522.30) - /unlink 55488 (time 1747400195.66 total 120.23 last 548.70) - /unlink 60000 (time 1747400204.50 total 129.08 last 510.00) - /unlink 64527 (time 1747400214.50 total 139.08 last 452.62) total: 64768 /unlink in 139.58 seconds: 464.03 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2687 (time 1747400246.04 total 10.00 last 268.68) - open/close 5198 (time 1747400256.04 total 20.00 last 251.08) - open/close 7762 (time 1747400266.04 total 30.00 last 256.35) - open/close 10000 (time 1747400274.58 total 38.54 last 262.24) - open/close 12297 (time 1747400284.58 total 48.54 last 229.67) - open/close 14863 (time 1747400294.58 total 58.55 last 256.45) - open/close 18068 (time 1747400304.59 total 68.55 last 320.47) - open/close 20000 (time 1747400310.89 total 74.85 last 306.55) - open/close 23105 (time 1747400320.89 total 84.85 last 310.49) - open/close 26073 (time 1747400330.89 total 94.85 last 296.80) - open/close 29389 (time 1747400340.89 total 104.85 last 331.55) - open/close 30000 (time 1747400343.15 total 107.11 last 270.48) - open/close 32730 (time 1747400353.15 total 117.11 last 272.89) - open/close 35720 (time 1747400363.15 total 127.11 last 299.00) - open/close 39156 (time 1747400373.15 total 137.11 last 343.57) - open/close 40000 (time 1747400376.29 total 140.25 last 268.85) - open/close 43537 (time 1747400386.29 total 150.26 last 353.64) - open/close 47071 (time 1747400396.29 total 160.26 last 353.40) - open/close 50000 (time 1747400405.08 total 169.04 last 333.23) - open/close 53809 (time 1747400415.09 total 179.05 last 380.86) - open/close 56939 (time 1747400425.09 total 189.05 last 312.93) - open/close 59786 (time 1747400435.09 total 199.05 last 284.61) - open/close 63520 (time 1747400445.09 total 209.05 last 373.35) total: 64768 open/close in 212.20 seconds: 305.23 ops/second - /unlink 7001 (time 1747400459.15 total 10.00 last 700.04) - /unlink 10000 (time 1747400462.62 total 13.47 last 863.95) - /unlink 18224 (time 1747400472.62 total 23.47 last 822.30) - /unlink 20000 (time 1747400474.67 total 25.53 last 865.60) - /unlink 29421 (time 1747400484.67 total 35.53 last 942.06) - /unlink 38494 (time 1747400494.67 total 45.53 last 907.30) - /unlink 47646 (time 1747400504.67 total 55.53 last 915.09) - /unlink 50000 (time 1747400507.61 total 58.46 last 802.03) - /unlink 58434 (time 1747400517.61 total 68.46 last 843.30) total: 64768 /unlink in 75.15 seconds: 861.81 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5112 (time 1747400560.77 total 10.00 last 511.13) - open/close 10000 (time 1747400569.40 total 18.63 last 566.16) - open/close 15289 (time 1747400579.40 total 28.64 last 528.86) - open/close 17673 (time 1747400589.41 total 38.64 last 238.22) - open/close 19172 (time 1747400599.43 total 48.66 last 149.70) - open/close 20000 (time 1747400605.62 total 54.85 last 133.61) - open/close 22127 (time 1747400615.62 total 64.86 last 212.65) - open/close 24626 (time 1747400625.63 total 74.86 last 249.85) - open/close 26455 (time 1747400635.64 total 84.87 last 182.69) - open/close 27699 (time 1747400645.65 total 94.88 last 124.29) - open/close 29453 (time 1747400655.65 total 104.88 last 175.35) - open/close 32416 (time 1747400665.65 total 114.88 last 296.24) - open/close 34854 (time 1747400675.65 total 124.88 last 243.77) - open/close 37603 (time 1747400685.66 total 134.89 last 274.76) - open/close 39765 (time 1747400695.66 total 144.89 last 216.10) - open/close 41577 (time 1747400705.67 total 154.90 last 181.09) - open/close 43209 (time 1747400715.67 total 164.91 last 163.12) - open/close 45710 (time 1747400725.68 total 174.91 last 250.03) - open/close 49785 (time 1747400735.68 total 184.91 last 407.37) - open/close 54974 (time 1747400745.68 total 194.91 last 518.82) - open/close 59256 (time 1747400755.68 total 204.91 last 428.14) - open/close 63835 (time 1747400765.68 total 214.92 last 457.82) total: 64768 open/close in 216.97 seconds: 298.51 ops/second - /unlink 7663 (time 1747400778.62 total 10.00 last 766.29) - /unlink 10000 (time 1747400781.03 total 12.41 last 969.57) - /unlink 19858 (time 1747400791.03 total 22.41 last 985.79) - /unlink 28685 (time 1747400801.03 total 32.41 last 882.65) - /unlink 38955 (time 1747400811.03 total 42.41 last 1026.90) - /unlink 48274 (time 1747400821.03 total 52.41 last 931.86) - /unlink 58074 (time 1747400831.03 total 62.41 last 979.92) total: 64768 /unlink in 68.38 seconds: 947.22 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5772 (time 1747400881.25 total 10.00 last 577.14) - open/close 10000 (time 1747400888.76 total 17.51 last 563.20) - open/close 15433 (time 1747400898.76 total 27.51 last 543.25) - open/close 20000 (time 1747400907.90 total 36.65 last 499.70) - open/close 25172 (time 1747400917.90 total 46.65 last 517.19) - open/close 30000 (time 1747400926.69 total 55.45 last 548.84) - open/close 35506 (time 1747400936.70 total 65.45 last 550.52) - open/close 40000 (time 1747400945.65 total 74.40 last 502.12) - open/close 44992 (time 1747400955.65 total 84.40 last 499.13) - open/close 49793 (time 1747400965.65 total 94.40 last 480.02) - open/close 54802 (time 1747400975.65 total 104.40 last 500.86) - open/close 59756 (time 1747400985.65 total 114.40 last 495.35) total: 64768 open/close in 124.29 seconds: 521.11 ops/second - /unlink 9260 (time 1747401006.36 total 10.00 last 925.95) - /unlink 18214 (time 1747401016.36 total 20.00 last 895.32) - /unlink 27368 (time 1747401026.36 total 30.00 last 915.37) - /unlink 30000 (time 1747401029.20 total 32.84 last 928.94) - /unlink 39194 (time 1747401039.20 total 42.84 last 919.38) - /unlink 48331 (time 1747401049.20 total 52.84 last 913.70) - /unlink 57151 (time 1747401059.20 total 62.84 last 881.96) - /unlink 60000 (time 1747401062.30 total 65.93 last 919.45) total: 64768 /unlink in 70.90 seconds: 913.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5620 (time 1747401111.48 total 10.00 last 561.96) - open/close 10000 (time 1747401118.78 total 17.30 last 600.11) - open/close 15206 (time 1747401128.78 total 27.30 last 520.54) - open/close 20000 (time 1747401138.01 total 36.53 last 519.51) - open/close 25224 (time 1747401148.01 total 46.53 last 522.33) - open/close 30000 (time 1747401157.35 total 55.87 last 511.43) - open/close 35144 (time 1747401167.35 total 65.87 last 514.35) - open/close 39929 (time 1747401177.35 total 75.87 last 478.42) - open/close 44868 (time 1747401187.35 total 85.87 last 493.88) - open/close 49719 (time 1747401197.35 total 95.87 last 485.01) - open/close 54227 (time 1747401207.35 total 105.87 last 450.73) - open/close 59809 (time 1747401217.35 total 115.88 last 558.16) total: 64768 open/close in 125.04 seconds: 517.97 ops/second - /unlink 9150 (time 1747401237.35 total 10.00 last 914.91) - /unlink 18245 (time 1747401247.35 total 20.00 last 909.46) - /unlink 20000 (time 1747401249.42 total 22.07 last 849.84) - /unlink 28907 (time 1747401259.42 total 32.07 last 890.56) - /unlink 37943 (time 1747401269.42 total 42.07 last 903.60) - /unlink 40000 (time 1747401271.82 total 44.47 last 854.88) - /unlink 48780 (time 1747401281.83 total 54.48 last 877.92) - /unlink 57948 (time 1747401291.83 total 64.48 last 916.75) - /unlink 60000 (time 1747401293.93 total 66.58 last 976.13) total: 64768 /unlink in 71.63 seconds: 904.17 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server LNET ready to unload unloading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: LNET ready to unload modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 PASS 106 (1750s) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 oleg136-server: oleg136-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1870 sec ========== 09:16:12 (1747401372)