== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:05:07 (1735099507) Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f) oleg439-server: oleg439-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg439-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg439-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2304 (time 1735099548.20 total 10.00 last 230.32) - open/close 4549 (time 1735099558.20 total 20.00 last 224.48) - open/close 6643 (time 1735099568.20 total 30.01 last 209.36) - open/close 8478 (time 1735099578.20 total 40.01 last 183.48) - open/close 10000 (time 1735099586.41 total 48.21 last 185.46) - open/close 11831 (time 1735099596.41 total 58.22 last 183.05) - open/close 13499 (time 1735099606.41 total 68.22 last 166.71) - open/close 15356 (time 1735099616.42 total 78.22 last 185.68) - open/close 16999 (time 1735099626.42 total 88.22 last 164.28) - open/close 18602 (time 1735099636.42 total 98.23 last 160.21) - open/close 20000 (time 1735099645.46 total 107.26 last 154.76) - open/close 21659 (time 1735099655.46 total 117.27 last 165.83) - open/close 23554 (time 1735099665.46 total 127.27 last 189.46) - open/close 25166 (time 1735099675.47 total 137.28 last 161.02) - open/close 26612 (time 1735099685.48 total 147.28 last 144.57) - open/close 28148 (time 1735099695.48 total 157.29 last 153.49) - open/close 29719 (time 1735099705.49 total 167.29 last 157.03) - open/close 31285 (time 1735099715.49 total 177.30 last 156.53) - open/close 32782 (time 1735099725.50 total 187.31 last 149.60) - open/close 34353 (time 1735099735.50 total 197.31 last 157.06) - open/close 36065 (time 1735099745.50 total 207.31 last 171.16) - open/close 37878 (time 1735099755.51 total 217.31 last 181.22) - open/close 40000 (time 1735099765.31 total 227.11 last 216.52) - open/close 42120 (time 1735099775.31 total 237.12 last 211.99) - open/close 44100 (time 1735099785.31 total 247.12 last 197.91) - open/close 46127 (time 1735099795.31 total 257.12 last 202.67) - open/close 48374 (time 1735099805.32 total 267.12 last 224.69) - open/close 50000 (time 1735099813.83 total 275.63 last 191.01) - open/close 52172 (time 1735099823.83 total 285.63 last 217.19) - open/close 54720 (time 1735099833.83 total 295.64 last 254.72) - open/close 57194 (time 1735099843.84 total 305.64 last 247.25) - open/close 59585 (time 1735099853.84 total 315.65 last 239.05) - open/close 62009 (time 1735099863.84 total 325.65 last 242.33) - open/close 64221 (time 1735099873.85 total 335.65 last 221.13) total: 64768 open/close in 338.02 seconds: 191.61 ops/second - /unlink 4023 (time 1735099887.31 total 10.00 last 402.23) - /unlink 7825 (time 1735099897.32 total 20.00 last 380.10) - /unlink 10000 (time 1735099903.31 total 26.00 last 362.95) - /unlink 13693 (time 1735099913.31 total 36.00 last 369.29) - /unlink 17041 (time 1735099923.31 total 46.00 last 334.73) - /unlink 20000 (time 1735099930.99 total 53.68 last 385.27) - /unlink 23792 (time 1735099940.99 total 63.68 last 379.15) - /unlink 26777 (time 1735099951.00 total 73.68 last 298.40) - /unlink 29367 (time 1735099961.00 total 83.69 last 258.95) - /unlink 30000 (time 1735099963.42 total 86.10 last 261.81) - /unlink 32799 (time 1735099973.42 total 96.11 last 279.87) - /unlink 36524 (time 1735099983.42 total 106.11 last 372.45) - /unlink 39532 (time 1735099993.42 total 116.11 last 300.65) - /unlink 40000 (time 1735099996.05 total 118.73 last 178.51) - /unlink 41376 (time 1735100006.05 total 128.74 last 137.54) - /unlink 42633 (time 1735100016.06 total 138.75 last 125.59) - /unlink 44310 (time 1735100026.06 total 148.75 last 167.70) - /unlink 46047 (time 1735100036.06 total 158.75 last 173.66) - /unlink 47896 (time 1735100046.06 total 168.75 last 184.90) - /unlink 49987 (time 1735100056.06 total 178.75 last 209.03) - /unlink 52068 (time 1735100066.07 total 188.76 last 208.00) - /unlink 54134 (time 1735100076.07 total 198.76 last 206.57) - /unlink 56142 (time 1735100086.07 total 208.76 last 200.79) - /unlink 58232 (time 1735100096.07 total 218.76 last 208.95) - /unlink 60000 (time 1735100103.97 total 226.66 last 223.91) - /unlink 63189 (time 1735100113.97 total 236.66 last 318.90) total: 64768 /unlink in 241.72 seconds: 267.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1672 (time 1735100146.46 total 10.00 last 167.12) - open/close 3451 (time 1735100156.46 total 20.01 last 177.83) - open/close 5031 (time 1735100166.46 total 30.01 last 157.95) - open/close 6771 (time 1735100176.47 total 40.01 last 173.98) - open/close 8172 (time 1735100186.47 total 50.02 last 139.99) - open/close 9479 (time 1735100196.48 total 60.02 last 130.66) - open/close 10000 (time 1735100200.40 total 63.95 last 132.86) - open/close 11335 (time 1735100210.41 total 73.95 last 133.42) - open/close 12710 (time 1735100220.41 total 83.95 last 137.50) - open/close 14116 (time 1735100230.41 total 93.96 last 140.54) - open/close 15679 (time 1735100240.41 total 103.96 last 156.30) - open/close 16927 (time 1735100250.42 total 113.96 last 124.73) - open/close 18257 (time 1735100260.42 total 123.97 last 132.91) - open/close 19571 (time 1735100270.43 total 133.98 last 131.27) - open/close 20000 (time 1735100273.47 total 137.01 last 141.44) - open/close 21370 (time 1735100283.47 total 147.02 last 136.92) - open/close 22750 (time 1735100293.47 total 157.02 last 137.99) - open/close 24184 (time 1735100303.47 total 167.02 last 143.36) - open/close 25815 (time 1735100313.48 total 177.02 last 163.08) - open/close 27570 (time 1735100323.48 total 187.03 last 175.44) - open/close 29218 (time 1735100333.48 total 197.03 last 164.75) - open/close 30000 (time 1735100337.33 total 200.87 last 203.41) - open/close 32330 (time 1735100347.33 total 210.88 last 232.89) - open/close 34779 (time 1735100357.33 total 220.88 last 244.90) - open/close 37901 (time 1735100367.33 total 230.88 last 312.14) - open/close 40000 (time 1735100375.21 total 238.76 last 266.51) - open/close 43062 (time 1735100385.21 total 248.76 last 306.16) - open/close 45743 (time 1735100395.21 total 258.76 last 268.09) - open/close 49006 (time 1735100405.21 total 268.76 last 326.19) - open/close 50000 (time 1735100408.05 total 271.59 last 351.05) - open/close 53213 (time 1735100418.05 total 281.59 last 321.27) - open/close 56690 (time 1735100428.05 total 291.60 last 347.60) - open/close 60000 (time 1735100437.01 total 300.55 last 369.60) - open/close 63055 (time 1735100447.01 total 310.55 last 305.48) total: 64768 open/close in 316.12 seconds: 204.89 ops/second - /unlink 6745 (time 1735100463.50 total 10.00 last 674.48) - /unlink 10000 (time 1735100467.97 total 14.47 last 727.81) - /unlink 15917 (time 1735100477.97 total 24.47 last 591.61) - /unlink 20000 (time 1735100483.80 total 30.31 last 700.13) - /unlink 27223 (time 1735100493.80 total 40.31 last 722.25) - /unlink 30000 (time 1735100498.17 total 44.67 last 635.83) - /unlink 37826 (time 1735100508.17 total 54.67 last 782.57) - /unlink 40000 (time 1735100510.84 total 57.34 last 814.56) - /unlink 48625 (time 1735100520.84 total 67.34 last 862.49) - /unlink 56547 (time 1735100530.84 total 77.34 last 792.16) - /unlink 60000 (time 1735100534.59 total 81.09 last 920.68) total: 64768 /unlink in 87.06 seconds: 743.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4432 (time 1735100571.77 total 10.00 last 443.19) - open/close 8780 (time 1735100581.77 total 20.00 last 434.71) - open/close 10000 (time 1735100584.03 total 22.27 last 538.37) - open/close 15177 (time 1735100594.03 total 32.27 last 517.67) - open/close 19815 (time 1735100604.04 total 42.27 last 463.64) - open/close 23954 (time 1735100614.04 total 52.27 last 413.81) - open/close 28839 (time 1735100624.04 total 62.27 last 488.47) - open/close 30000 (time 1735100626.63 total 64.86 last 448.52) - open/close 34877 (time 1735100636.63 total 74.86 last 487.67) - open/close 39610 (time 1735100646.63 total 84.86 last 473.26) - open/close 44351 (time 1735100656.63 total 94.87 last 474.05) - open/close 48878 (time 1735100666.63 total 104.87 last 452.70) - open/close 50000 (time 1735100669.56 total 107.79 last 383.18) - open/close 54463 (time 1735100679.56 total 117.80 last 446.23) - open/close 59458 (time 1735100689.56 total 127.80 last 499.48) - open/close 64013 (time 1735100699.56 total 137.80 last 455.47) total: 64768 open/close in 139.49 seconds: 464.34 ops/second - /unlink 8204 (time 1735100712.12 total 10.00 last 820.35) - /unlink 10000 (time 1735100714.66 total 12.54 last 706.84) - /unlink 17238 (time 1735100724.66 total 22.54 last 723.74) - /unlink 20000 (time 1735100728.16 total 26.05 last 787.74) - /unlink 28612 (time 1735100738.16 total 36.05 last 861.15) - /unlink 36440 (time 1735100748.16 total 46.05 last 782.78) - /unlink 40000 (time 1735100752.56 total 50.44 last 810.17) - /unlink 47516 (time 1735100762.56 total 60.44 last 751.60) - /unlink 50000 (time 1735100766.05 total 63.94 last 711.41) - /unlink 58402 (time 1735100776.05 total 73.94 last 840.14) total: 64768 /unlink in 81.47 seconds: 794.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5102 (time 1735100811.42 total 10.00 last 510.12) - open/close 9157 (time 1735100821.42 total 20.00 last 405.44) - open/close 13206 (time 1735100831.42 total 30.00 last 404.88) - open/close 17446 (time 1735100841.42 total 40.01 last 423.92) - open/close 20000 (time 1735100847.26 total 45.84 last 437.55) - open/close 24471 (time 1735100857.26 total 55.84 last 447.01) - open/close 27923 (time 1735100867.26 total 65.85 last 345.14) - open/close 30000 (time 1735100873.55 total 72.14 last 330.18) - open/close 33767 (time 1735100883.55 total 82.14 last 376.64) - open/close 37732 (time 1735100893.55 total 92.14 last 396.48) - open/close 40000 (time 1735100899.23 total 97.81 last 399.64) - open/close 43475 (time 1735100909.23 total 107.82 last 347.43) - open/close 47796 (time 1735100919.23 total 117.82 last 432.04) - open/close 50000 (time 1735100924.03 total 122.61 last 459.63) - open/close 55016 (time 1735100934.03 total 132.61 last 501.55) - open/close 60000 (time 1735100943.13 total 141.71 last 547.84) total: 64768 open/close in 150.18 seconds: 431.26 ops/second - /unlink 8547 (time 1735100962.37 total 10.00 last 854.62) - /unlink 17516 (time 1735100972.37 total 20.00 last 896.84) - /unlink 20000 (time 1735100975.21 total 22.84 last 874.86) - /unlink 28854 (time 1735100985.21 total 32.84 last 885.39) - /unlink 36803 (time 1735100995.21 total 42.84 last 794.89) - /unlink 40000 (time 1735100999.86 total 47.50 last 686.86) - /unlink 47969 (time 1735101009.86 total 57.50 last 796.89) - /unlink 50000 (time 1735101012.39 total 60.02 last 804.88) - /unlink 58905 (time 1735101022.39 total 70.02 last 890.41) total: 64768 /unlink in 76.95 seconds: 841.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4450 (time 1735101062.28 total 10.00 last 444.91) - open/close 8736 (time 1735101072.28 total 20.00 last 428.59) - open/close 10000 (time 1735101075.36 total 23.07 last 411.45) - open/close 13810 (time 1735101085.36 total 33.08 last 380.95) - open/close 19048 (time 1735101095.36 total 43.08 last 523.77) - open/close 24005 (time 1735101105.36 total 53.08 last 495.65) - open/close 28145 (time 1735101115.36 total 63.08 last 413.97) - open/close 30000 (time 1735101120.03 total 67.75 last 396.86) - open/close 33790 (time 1735101130.03 total 77.75 last 378.95) - open/close 37603 (time 1735101140.04 total 87.75 last 381.26) - open/close 40000 (time 1735101145.13 total 92.85 last 470.34) - open/close 44536 (time 1735101155.13 total 102.85 last 453.58) - open/close 47030 (time 1735101165.14 total 112.85 last 249.32) - open/close 48797 (time 1735101175.14 total 122.86 last 176.69) - open/close 50000 (time 1735101178.41 total 126.13 last 367.78) - open/close 53238 (time 1735101188.41 total 136.13 last 323.80) - open/close 56484 (time 1735101198.41 total 146.13 last 324.60) - open/close 58480 (time 1735101208.42 total 156.14 last 199.41) - open/close 60000 (time 1735101215.06 total 162.78 last 228.67) - open/close 63059 (time 1735101225.07 total 172.78 last 305.84) total: 64768 open/close in 176.86 seconds: 366.20 ops/second - /unlink 7749 (time 1735101240.13 total 10.00 last 774.62) - /unlink 10000 (time 1735101244.30 total 14.18 last 539.01) - /unlink 15808 (time 1735101254.30 total 24.18 last 580.80) - /unlink 20000 (time 1735101260.97 total 30.84 last 629.07) - /unlink 26972 (time 1735101270.97 total 40.84 last 697.17) - /unlink 30000 (time 1735101275.09 total 44.97 last 733.74) - /unlink 36256 (time 1735101285.09 total 54.97 last 625.55) - /unlink 40000 (time 1735101292.22 total 62.09 last 525.75) - /unlink 46036 (time 1735101302.22 total 72.09 last 603.59) - /unlink 50000 (time 1735101307.47 total 77.35 last 754.52) - /unlink 57341 (time 1735101317.47 total 87.35 last 734.09) - /unlink 60000 (time 1735101322.01 total 91.89 last 585.34) total: 64768 /unlink in 100.36 seconds: 645.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2