-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:14 EDT 2025 oleg419-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:47:06 (1747399626) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 911 (time 1747399702.68 total 10.01 last 91.00) - open/close 1925 (time 1747399712.69 total 20.01 last 101.36) - open/close 3039 (time 1747399722.69 total 30.02 last 111.38) - open/close 4531 (time 1747399732.69 total 40.02 last 149.14) - open/close 5897 (time 1747399742.70 total 50.02 last 136.57) - open/close 7195 (time 1747399752.70 total 60.03 last 129.72) - open/close 8511 (time 1747399762.71 total 70.04 last 131.51) - open/close 9688 (time 1747399772.71 total 80.04 last 117.65) - open/close 10000 (time 1747399774.94 total 82.26 last 140.37) - open/close 11293 (time 1747399784.94 total 92.27 last 129.25) - open/close 12526 (time 1747399794.94 total 102.27 last 123.28) - open/close 13841 (time 1747399804.94 total 112.27 last 131.46) - open/close 15163 (time 1747399814.95 total 122.28 last 132.15) - open/close 16522 (time 1747399824.95 total 132.28 last 135.83) - open/close 17835 (time 1747399834.95 total 142.28 last 131.28) - open/close 19126 (time 1747399844.96 total 152.29 last 129.06) - open/close 20000 (time 1747399851.62 total 158.94 last 131.25) - open/close 21349 (time 1747399861.62 total 168.94 last 134.90) - open/close 23170 (time 1747399871.62 total 178.95 last 182.03) - open/close 25234 (time 1747399881.62 total 188.95 last 206.35) - open/close 27277 (time 1747399891.62 total 198.95 last 204.29) - open/close 29346 (time 1747399901.63 total 208.95 last 206.83) - open/close 30000 (time 1747399904.85 total 212.18 last 203.00) - open/close 32069 (time 1747399914.85 total 222.18 last 206.87) - open/close 34216 (time 1747399924.85 total 232.18 last 214.64) - open/close 36028 (time 1747399934.86 total 242.18 last 181.14) - open/close 37926 (time 1747399944.86 total 252.18 last 189.76) - open/close 39962 (time 1747399954.86 total 262.19 last 203.53) - open/close 41900 (time 1747399964.86 total 272.19 last 193.80) - open/close 43837 (time 1747399974.86 total 282.19 last 193.65) - open/close 45971 (time 1747399984.86 total 292.19 last 213.39) - open/close 47591 (time 1747399994.87 total 302.19 last 161.96) - open/close 49204 (time 1747400004.87 total 312.20 last 161.22) - open/close 50000 (time 1747400009.58 total 316.91 last 168.99) - open/close 51860 (time 1747400019.59 total 326.91 last 185.95) - open/close 53396 (time 1747400029.59 total 336.91 last 153.56) - open/close 54819 (time 1747400039.59 total 346.92 last 142.20) - open/close 56360 (time 1747400049.60 total 356.93 last 154.01) - open/close 58107 (time 1747400059.60 total 366.93 last 174.68) - open/close 59945 (time 1747400069.61 total 376.93 last 183.73) - open/close 61709 (time 1747400079.61 total 386.93 last 176.37) - open/close 63577 (time 1747400089.61 total 396.94 last 186.74) total: 64768 open/close in 402.29 seconds: 161.00 ops/second - /unlink 3880 (time 1747400106.01 total 10.00 last 387.99) - /unlink 7624 (time 1747400116.01 total 20.00 last 374.29) - /unlink 10000 (time 1747400122.82 total 26.81 last 349.12) - /unlink 13512 (time 1747400132.82 total 36.81 last 351.11) - /unlink 17325 (time 1747400142.82 total 46.81 last 381.24) - /unlink 20000 (time 1747400149.36 total 53.35 last 408.99) - /unlink 24293 (time 1747400159.37 total 63.36 last 429.22) - /unlink 28452 (time 1747400169.37 total 73.36 last 415.87) - /unlink 30000 (time 1747400173.23 total 77.22 last 401.08) - /unlink 34296 (time 1747400183.23 total 87.22 last 429.60) - /unlink 38652 (time 1747400193.23 total 97.22 last 435.50) - /unlink 40000 (time 1747400196.33 total 100.32 last 434.99) - /unlink 43685 (time 1747400206.33 total 110.32 last 368.43) - /unlink 47566 (time 1747400216.33 total 120.32 last 388.07) - /unlink 50000 (time 1747400222.10 total 126.09 last 421.96) - /unlink 53921 (time 1747400232.10 total 136.09 last 392.02) - /unlink 57712 (time 1747400242.10 total 146.09 last 379.07) - /unlink 60000 (time 1747400248.12 total 152.11 last 380.31) - /unlink 64548 (time 1747400258.12 total 162.11 last 454.72) total: 64768 /unlink in 162.72 seconds: 398.04 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2811 (time 1747400286.37 total 10.00 last 281.09) - open/close 5239 (time 1747400296.37 total 20.00 last 242.77) - open/close 7940 (time 1747400306.37 total 30.00 last 270.04) - open/close 10000 (time 1747400314.05 total 37.68 last 268.29) - open/close 12811 (time 1747400324.06 total 47.68 last 281.02) - open/close 16083 (time 1747400334.06 total 57.69 last 327.17) - open/close 19911 (time 1747400344.06 total 67.69 last 382.75) - open/close 22727 (time 1747400354.06 total 77.69 last 281.53) - open/close 25232 (time 1747400364.06 total 87.69 last 250.48) - open/close 27983 (time 1747400374.06 total 97.69 last 275.03) - open/close 30000 (time 1747400382.10 total 105.73 last 251.09) - open/close 32676 (time 1747400392.10 total 115.73 last 267.57) - open/close 35513 (time 1747400402.10 total 125.73 last 283.66) - open/close 37939 (time 1747400412.10 total 135.73 last 242.56) - open/close 40000 (time 1747400420.71 total 144.34 last 239.42) - open/close 42899 (time 1747400430.71 total 154.34 last 289.80) - open/close 45895 (time 1747400440.72 total 164.34 last 299.50) - open/close 49313 (time 1747400450.72 total 174.35 last 341.73) - open/close 50000 (time 1747400452.73 total 176.36 last 341.38) - open/close 53943 (time 1747400462.73 total 186.36 last 394.26) - open/close 57243 (time 1747400472.73 total 196.36 last 329.96) - open/close 60000 (time 1747400479.87 total 203.50 last 386.27) - open/close 64226 (time 1747400489.87 total 213.50 last 422.58) total: 64768 open/close in 214.84 seconds: 301.46 ops/second - /unlink 7083 (time 1747400502.11 total 10.00 last 708.26) - /unlink 10000 (time 1747400505.75 total 13.64 last 801.80) - /unlink 17190 (time 1747400515.75 total 23.64 last 718.97) - /unlink 20000 (time 1747400520.46 total 28.35 last 596.26) - /unlink 26552 (time 1747400530.46 total 38.35 last 655.20) - /unlink 30000 (time 1747400535.89 total 43.78 last 635.47) - /unlink 36834 (time 1747400545.89 total 53.78 last 683.38) - /unlink 40000 (time 1747400549.98 total 57.88 last 772.39) - /unlink 47818 (time 1747400559.98 total 67.88 last 781.80) - /unlink 50000 (time 1747400562.61 total 70.50 last 831.94) - /unlink 58469 (time 1747400572.61 total 80.50 last 846.79) total: 64768 /unlink in 88.06 seconds: 735.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2191 (time 1747400612.09 total 10.01 last 218.94) - open/close 4362 (time 1747400622.09 total 20.01 last 217.07) - open/close 6666 (time 1747400632.09 total 30.01 last 230.35) - open/close 8504 (time 1747400642.10 total 40.01 last 183.77) - open/close 10000 (time 1747400649.69 total 47.61 last 196.99) - open/close 12001 (time 1747400659.69 total 57.61 last 200.03) - open/close 14888 (time 1747400669.69 total 67.61 last 288.65) - open/close 18202 (time 1747400679.70 total 77.61 last 331.33) - open/close 20000 (time 1747400684.86 total 82.78 last 348.05) - open/close 22697 (time 1747400694.86 total 92.78 last 269.65) - open/close 24712 (time 1747400704.87 total 102.78 last 201.45) - open/close 26749 (time 1747400714.87 total 112.78 last 203.67) - open/close 28411 (time 1747400724.87 total 122.79 last 166.19) - open/close 30000 (time 1747400731.57 total 129.48 last 237.19) - open/close 33024 (time 1747400741.57 total 139.49 last 302.25) - open/close 35904 (time 1747400751.58 total 149.49 last 287.91) - open/close 38710 (time 1747400761.58 total 159.49 last 280.59) - open/close 40000 (time 1747400765.67 total 163.58 last 315.42) - open/close 42904 (time 1747400775.67 total 173.58 last 290.39) - open/close 47324 (time 1747400785.67 total 183.58 last 441.99) - open/close 50000 (time 1747400792.48 total 190.40 last 392.59) - open/close 53922 (time 1747400802.49 total 200.40 last 392.14) - open/close 57871 (time 1747400812.49 total 210.40 last 394.80) - open/close 60000 (time 1747400817.85 total 215.76 last 397.15) - open/close 63647 (time 1747400827.85 total 225.76 last 364.69) total: 64768 open/close in 228.79 seconds: 283.09 ops/second - /unlink 7003 (time 1747400841.78 total 10.00 last 700.25) - /unlink 10000 (time 1747400846.40 total 14.63 last 647.77) - /unlink 16659 (time 1747400856.41 total 24.63 last 665.83) - /unlink 20000 (time 1747400861.06 total 29.28 last 718.53) - /unlink 27056 (time 1747400871.06 total 39.28 last 705.52) - /unlink 30000 (time 1747400875.32 total 43.54 last 691.28) - /unlink 37641 (time 1747400885.32 total 53.54 last 764.06) - /unlink 40000 (time 1747400888.70 total 56.93 last 696.54) - /unlink 47260 (time 1747400898.70 total 66.93 last 725.93) - /unlink 50000 (time 1747400902.00 total 70.22 last 831.92) - /unlink 58440 (time 1747400912.00 total 80.22 last 843.93) total: 64768 /unlink in 87.41 seconds: 740.95 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4978 (time 1747400952.18 total 10.00 last 497.72) - open/close 10000 (time 1747400961.58 total 19.40 last 534.48) - open/close 15323 (time 1747400971.58 total 29.40 last 532.24) - open/close 20000 (time 1747400980.38 total 38.20 last 531.43) - open/close 24952 (time 1747400990.38 total 48.20 last 495.10) - open/close 29402 (time 1747401000.38 total 58.20 last 444.90) - open/close 33907 (time 1747401010.38 total 68.20 last 450.45) - open/close 38394 (time 1747401020.39 total 78.21 last 448.57) - open/close 40000 (time 1747401024.55 total 82.37 last 385.87) - open/close 44393 (time 1747401034.55 total 92.37 last 439.26) - open/close 48855 (time 1747401044.55 total 102.37 last 446.18) - open/close 50000 (time 1747401046.85 total 104.67 last 498.61) - open/close 55088 (time 1747401056.85 total 114.67 last 508.74) - open/close 60000 (time 1747401065.64 total 123.46 last 558.84) - open/close 64599 (time 1747401075.64 total 133.46 last 459.83) total: 64768 open/close in 133.83 seconds: 483.95 ops/second - /unlink 7951 (time 1747401086.94 total 10.00 last 794.99) - /unlink 10000 (time 1747401089.54 total 12.60 last 787.14) - /unlink 18336 (time 1747401099.54 total 22.60 last 833.55) - /unlink 20000 (time 1747401101.60 total 24.66 last 809.41) - /unlink 27717 (time 1747401111.60 total 34.66 last 771.61) - /unlink 30000 (time 1747401115.18 total 38.25 last 636.79) - /unlink 38556 (time 1747401125.18 total 48.25 last 855.55) - /unlink 47258 (time 1747401135.18 total 58.25 last 870.12) - /unlink 50000 (time 1747401138.03 total 61.09 last 964.07) - /unlink 60000 (time 1747401147.89 total 70.96 last 1013.63) total: 64768 /unlink in 75.59 seconds: 856.87 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4107 (time 1747401187.81 total 10.00 last 410.66) - open/close 9328 (time 1747401197.81 total 20.00 last 521.99) - open/close 14300 (time 1747401207.81 total 30.00 last 497.14) - open/close 19938 (time 1747401217.81 total 40.00 last 563.80) - open/close 25474 (time 1747401227.81 total 50.00 last 553.58) - open/close 30000 (time 1747401236.76 total 58.95 last 505.73) - open/close 35493 (time 1747401246.76 total 68.95 last 549.28) - open/close 40000 (time 1747401255.34 total 77.53 last 525.40) - open/close 45396 (time 1747401265.34 total 87.53 last 539.56) - open/close 50000 (time 1747401274.67 total 96.87 last 493.28) - open/close 55181 (time 1747401284.67 total 106.87 last 518.04) - open/close 59588 (time 1747401294.67 total 116.87 last 440.69) - open/close 64088 (time 1747401304.67 total 126.87 last 449.97) total: 64768 open/close in 128.18 seconds: 505.30 ops/second - /unlink 9598 (time 1747401316.79 total 10.00 last 959.74) - /unlink 19268 (time 1747401326.79 total 20.00 last 966.93) - /unlink 28032 (time 1747401336.79 total 30.00 last 876.35) - /unlink 30000 (time 1747401339.10 total 32.32 last 850.37) - /unlink 38008 (time 1747401349.11 total 42.32 last 800.74) - /unlink 40000 (time 1747401351.65 total 44.86 last 783.75) - /unlink 48382 (time 1747401361.65 total 54.86 last 838.11) - /unlink 56532 (time 1747401371.65 total 64.86 last 814.97) - /unlink 60000 (time 1747401375.54 total 68.75 last 891.37) total: 64768 /unlink in 73.65 seconds: 879.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server LNET ready to unload unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: LNET ready to unload modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 106 (1799s) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1928 sec ========== 09:17:23 (1747401443)