== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:37:09 (1731749829) Stopping clients: oleg354-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg354-client.virtnet /mnt/lustre2 (opts:-f) oleg354-server: oleg354-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: detected 4 online CPUs by sysfs oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg354-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg354-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg354-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1658 (time 1731749864.11 total 10.00 last 165.73) - open/close 2975 (time 1731749874.12 total 20.01 last 131.63) - open/close 5351 (time 1731749884.12 total 30.01 last 237.56) - open/close 7142 (time 1731749894.12 total 40.01 last 179.08) - open/close 9179 (time 1731749904.12 total 50.01 last 203.66) - open/close 10000 (time 1731749908.05 total 53.94 last 209.23) - open/close 11663 (time 1731749918.05 total 63.94 last 166.24) - open/close 13293 (time 1731749928.06 total 73.95 last 162.92) - open/close 15220 (time 1731749938.06 total 83.95 last 192.65) - open/close 17288 (time 1731749948.06 total 93.95 last 206.80) - open/close 19802 (time 1731749958.06 total 103.95 last 251.36) - open/close 21839 (time 1731749968.06 total 113.95 last 203.68) - open/close 24104 (time 1731749978.07 total 123.96 last 226.44) - open/close 25864 (time 1731749988.07 total 133.96 last 175.91) - open/close 27399 (time 1731749998.07 total 143.96 last 153.45) - open/close 28848 (time 1731750008.08 total 153.97 last 144.85) - open/close 30000 (time 1731750016.14 total 162.03 last 142.89) - open/close 31405 (time 1731750026.14 total 172.03 last 140.46) - open/close 32901 (time 1731750036.15 total 182.04 last 149.52) - open/close 34319 (time 1731750046.15 total 192.04 last 141.73) - open/close 35707 (time 1731750056.15 total 202.04 last 138.78) - open/close 37207 (time 1731750066.16 total 212.05 last 149.96) - open/close 38740 (time 1731750076.16 total 222.05 last 153.28) - open/close 40000 (time 1731750083.86 total 229.75 last 163.54) - open/close 41931 (time 1731750093.86 total 239.75 last 193.05) - open/close 44301 (time 1731750103.87 total 249.76 last 236.96) - open/close 46822 (time 1731750113.87 total 259.76 last 252.05) - open/close 50000 (time 1731750122.92 total 268.81 last 350.98) - open/close 53763 (time 1731750132.92 total 278.81 last 376.29) - open/close 56956 (time 1731750142.93 total 288.82 last 319.23) - open/close 60000 (time 1731750151.45 total 297.34 last 357.14) - open/close 63280 (time 1731750161.46 total 307.35 last 327.78) total: 64768 open/close in 312.17 seconds: 207.48 ops/second - /unlink 6396 (time 1731750177.21 total 10.00 last 639.57) - /unlink 10000 (time 1731750184.21 total 17.01 last 514.43) - /unlink 14562 (time 1731750194.22 total 27.01 last 456.01) - /unlink 18609 (time 1731750204.22 total 37.01 last 404.67) - /unlink 20000 (time 1731750208.01 total 40.81 last 366.45) - /unlink 23736 (time 1731750218.01 total 50.81 last 373.58) - /unlink 27352 (time 1731750228.02 total 60.81 last 361.56) - /unlink 30000 (time 1731750234.72 total 67.51 last 395.14) - /unlink 33827 (time 1731750244.72 total 77.51 last 382.70) - /unlink 37821 (time 1731750254.72 total 87.51 last 399.33) - /unlink 40000 (time 1731750260.12 total 92.91 last 403.74) - /unlink 44036 (time 1731750270.12 total 102.91 last 403.58) - /unlink 47785 (time 1731750280.12 total 112.91 last 374.88) - /unlink 50000 (time 1731750286.10 total 118.90 last 370.07) - /unlink 54508 (time 1731750296.10 total 128.90 last 450.79) - /unlink 59519 (time 1731750306.10 total 138.90 last 501.06) - /unlink 64334 (time 1731750316.10 total 148.90 last 481.45) total: 64768 /unlink in 149.78 seconds: 432.43 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2531 (time 1731750342.66 total 10.00 last 253.04) - open/close 5757 (time 1731750352.66 total 20.00 last 322.58) - open/close 8599 (time 1731750362.66 total 30.00 last 284.17) - open/close 10000 (time 1731750367.81 total 35.15 last 272.45) - open/close 12875 (time 1731750377.81 total 45.15 last 287.41) - open/close 15815 (time 1731750387.81 total 55.15 last 293.92) - open/close 18448 (time 1731750397.81 total 65.15 last 263.28) - open/close 20000 (time 1731750403.55 total 70.89 last 270.34) - open/close 22634 (time 1731750413.55 total 80.89 last 263.40) - open/close 25139 (time 1731750423.56 total 90.90 last 250.39) - open/close 28101 (time 1731750433.56 total 100.90 last 296.16) - open/close 30000 (time 1731750440.66 total 108.01 last 267.26) - open/close 32665 (time 1731750450.67 total 118.01 last 266.48) - open/close 35272 (time 1731750460.67 total 128.01 last 260.69) - open/close 37786 (time 1731750470.67 total 138.01 last 251.34) - open/close 40000 (time 1731750478.68 total 146.02 last 276.33) - open/close 43620 (time 1731750488.68 total 156.02 last 362.00) - open/close 47048 (time 1731750498.68 total 166.02 last 342.78) - open/close 50000 (time 1731750507.65 total 174.99 last 329.18) - open/close 53119 (time 1731750517.65 total 184.99 last 311.79) - open/close 56361 (time 1731750527.65 total 194.99 last 324.18) - open/close 59419 (time 1731750537.65 total 205.00 last 305.75) - open/close 63351 (time 1731750547.66 total 215.00 last 393.16) total: 64768 open/close in 218.91 seconds: 295.86 ops/second - /unlink 4747 (time 1731750562.59 total 10.00 last 474.59) - /unlink 10000 (time 1731750570.39 total 17.80 last 673.37) - /unlink 17463 (time 1731750580.39 total 27.80 last 746.25) - /unlink 20000 (time 1731750584.30 total 31.71 last 649.82) - /unlink 27410 (time 1731750594.30 total 41.71 last 740.92) - /unlink 30000 (time 1731750597.69 total 45.10 last 764.58) - /unlink 36843 (time 1731750607.69 total 55.10 last 684.29) - /unlink 40000 (time 1731750611.67 total 59.08 last 791.93) - /unlink 47429 (time 1731750621.67 total 69.08 last 742.87) - /unlink 50000 (time 1731750624.67 total 72.08 last 857.29) - /unlink 59878 (time 1731750634.67 total 82.08 last 987.80) total: 64768 /unlink in 86.99 seconds: 744.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5544 (time 1731750673.09 total 10.00 last 554.32) - open/close 10000 (time 1731750682.12 total 19.04 last 493.13) - open/close 15605 (time 1731750692.12 total 29.04 last 560.42) - open/close 20000 (time 1731750699.50 total 36.42 last 595.57) - open/close 25414 (time 1731750709.50 total 46.42 last 541.36) - open/close 30000 (time 1731750717.33 total 54.25 last 585.82) - open/close 35433 (time 1731750727.34 total 64.25 last 543.19) - open/close 40000 (time 1731750736.63 total 73.54 last 491.40) - open/close 44908 (time 1731750746.63 total 83.54 last 490.73) - open/close 49395 (time 1731750756.63 total 93.54 last 448.69) - open/close 54490 (time 1731750766.63 total 103.55 last 509.47) - open/close 59066 (time 1731750776.63 total 113.55 last 457.60) - open/close 60000 (time 1731750778.80 total 115.71 last 430.59) total: 64768 open/close in 125.04 seconds: 517.98 ops/second - /unlink 8528 (time 1731750798.97 total 10.00 last 852.73) - /unlink 17560 (time 1731750808.97 total 20.00 last 903.19) - /unlink 20000 (time 1731750811.59 total 22.62 last 931.01) - /unlink 29004 (time 1731750821.59 total 32.62 last 900.31) - /unlink 37860 (time 1731750831.60 total 42.62 last 885.55) - /unlink 40000 (time 1731750834.10 total 45.12 last 855.63) - /unlink 48918 (time 1731750844.10 total 55.13 last 891.74) - /unlink 57365 (time 1731750854.10 total 65.13 last 844.62) - /unlink 60000 (time 1731750857.20 total 68.23 last 848.96) total: 64768 /unlink in 73.57 seconds: 880.41 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4870 (time 1731750898.66 total 10.00 last 486.91) - open/close 9229 (time 1731750908.70 total 20.04 last 434.30) - open/close 14414 (time 1731750918.70 total 30.04 last 518.44) - open/close 19094 (time 1731750928.70 total 40.04 last 467.99) - open/close 20000 (time 1731750930.83 total 42.17 last 425.37) - open/close 24684 (time 1731750940.83 total 52.18 last 468.15) - open/close 29409 (time 1731750950.83 total 62.18 last 472.48) - open/close 34583 (time 1731750960.84 total 72.18 last 517.28) - open/close 39795 (time 1731750970.84 total 82.18 last 521.19) - open/close 44591 (time 1731750980.84 total 92.18 last 479.51) - open/close 49322 (time 1731750990.84 total 102.18 last 473.02) - open/close 53774 (time 1731751000.84 total 112.18 last 445.14) - open/close 58305 (time 1731751010.84 total 122.19 last 453.01) - open/close 60000 (time 1731751014.75 total 126.09 last 433.95) - open/close 64682 (time 1731751024.75 total 136.09 last 468.12) total: 64768 open/close in 136.27 seconds: 475.28 ops/second - /unlink 8549 (time 1731751035.83 total 10.00 last 854.89) - /unlink 17049 (time 1731751045.84 total 20.00 last 849.95) - /unlink 20000 (time 1731751049.40 total 23.57 last 827.03) - /unlink 27893 (time 1731751059.40 total 33.57 last 789.29) - /unlink 30000 (time 1731751062.09 total 36.25 last 784.65) - /unlink 38078 (time 1731751072.09 total 46.25 last 807.78) - /unlink 40000 (time 1731751074.64 total 48.80 last 753.79) - /unlink 47639 (time 1731751084.64 total 58.80 last 763.89) - /unlink 50000 (time 1731751087.58 total 61.75 last 802.71) - /unlink 58457 (time 1731751097.58 total 71.75 last 845.56) total: 64768 /unlink in 79.01 seconds: 819.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4680 (time 1731751134.32 total 10.00 last 467.89) - open/close 9772 (time 1731751144.32 total 20.00 last 509.19) - open/close 14840 (time 1731751154.33 total 30.01 last 506.66) - open/close 19965 (time 1731751164.33 total 40.01 last 512.43) - open/close 25129 (time 1731751174.33 total 50.01 last 516.35) - open/close 30000 (time 1731751183.98 total 59.66 last 504.79) - open/close 35104 (time 1731751193.98 total 69.66 last 510.36) - open/close 40000 (time 1731751203.43 total 79.11 last 518.01) - open/close 45131 (time 1731751213.43 total 89.11 last 513.08) - open/close 49999 (time 1731751223.43 total 99.11 last 486.71) - open/close 54923 (time 1731751233.43 total 109.11 last 492.36) - open/close 59980 (time 1731751243.44 total 119.11 last 505.63) total: 64768 open/close in 128.61 seconds: 503.60 ops/second - /unlink 8915 (time 1731751263.81 total 10.00 last 891.42) - /unlink 17867 (time 1731751273.81 total 20.00 last 895.10) - /unlink 20000 (time 1731751276.16 total 22.35 last 907.88) - /unlink 28942 (time 1731751286.16 total 32.35 last 894.11) - /unlink 37868 (time 1731751296.16 total 42.35 last 892.53) - /unlink 40000 (time 1731751299.13 total 45.32 last 718.19) - /unlink 48806 (time 1731751309.13 total 55.32 last 880.58) - /unlink 57571 (time 1731751319.13 total 65.32 last 876.49) - /unlink 60000 (time 1731751321.91 total 68.11 last 871.99) total: 64768 /unlink in 73.34 seconds: 883.15 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2