== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:37:45 (1721878665) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) oleg221-server: oleg221-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock oleg221-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5354 (time 1721878698.52 total 10.00 last 535.34) - open/close 10000 (time 1721878707.23 total 18.70 last 533.89) - open/close 15364 (time 1721878717.23 total 28.70 last 536.36) - open/close 20000 (time 1721878725.88 total 37.36 last 535.78) - open/close 25353 (time 1721878735.88 total 47.36 last 535.29) - open/close 30000 (time 1721878744.74 total 56.22 last 524.34) - open/close 35240 (time 1721878754.74 total 66.22 last 523.99) - open/close 40000 (time 1721878763.73 total 75.21 last 529.65) - open/close 45114 (time 1721878773.73 total 85.21 last 511.32) - open/close 49430 (time 1721878783.73 total 95.21 last 431.57) - open/close 53784 (time 1721878793.73 total 105.21 last 435.39) - open/close 57155 (time 1721878803.73 total 115.21 last 337.04) - open/close 60000 (time 1721878812.58 total 124.06 last 321.49) - open/close 64597 (time 1721878822.58 total 134.06 last 459.69) total: 64768 open/close in 134.46 seconds: 481.70 ops/second - /unlink 7684 (time 1721878833.80 total 10.00 last 768.34) - /unlink 10000 (time 1721878836.48 total 12.68 last 865.17) - /unlink 18720 (time 1721878846.48 total 22.68 last 871.94) - /unlink 27288 (time 1721878856.48 total 32.68 last 856.79) - /unlink 30000 (time 1721878859.62 total 35.82 last 864.20) - /unlink 38778 (time 1721878869.62 total 45.82 last 877.71) - /unlink 47378 (time 1721878879.62 total 55.82 last 859.91) - /unlink 50000 (time 1721878882.74 total 58.94 last 841.04) - /unlink 58556 (time 1721878892.74 total 68.94 last 855.57) total: 64768 /unlink in 77.61 seconds: 834.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5965 (time 1721878932.56 total 10.00 last 596.41) - open/close 9673 (time 1721878942.57 total 20.00 last 370.73) - open/close 14531 (time 1721878952.57 total 30.00 last 485.75) - open/close 20000 (time 1721878962.04 total 39.48 last 577.00) - open/close 24910 (time 1721878972.05 total 49.48 last 490.92) - open/close 29515 (time 1721878982.05 total 59.49 last 460.43) - open/close 33516 (time 1721878992.05 total 69.49 last 400.06) - open/close 38137 (time 1721879002.05 total 79.49 last 462.04) - open/close 40000 (time 1721879005.93 total 83.37 last 480.38) - open/close 44753 (time 1721879015.93 total 93.37 last 475.29) - open/close 48754 (time 1721879025.93 total 103.37 last 400.07) - open/close 50000 (time 1721879029.24 total 106.68 last 376.18) - open/close 51990 (time 1721879039.25 total 116.69 last 198.89) - open/close 53291 (time 1721879049.25 total 126.69 last 130.04) - open/close 55211 (time 1721879059.26 total 136.70 last 191.90) - open/close 57068 (time 1721879069.26 total 146.70 last 185.64) - open/close 59011 (time 1721879079.26 total 156.70 last 194.29) - open/close 60000 (time 1721879085.29 total 162.73 last 164.07) - open/close 61976 (time 1721879095.29 total 172.73 last 197.51) - open/close 64444 (time 1721879105.29 total 182.73 last 246.77) total: 64768 open/close in 183.79 seconds: 352.41 ops/second - /unlink 5332 (time 1721879117.35 total 10.00 last 533.12) - /unlink 10000 (time 1721879126.12 total 18.78 last 531.89) - /unlink 15286 (time 1721879136.12 total 28.78 last 528.54) - /unlink 19220 (time 1721879146.12 total 38.78 last 393.34) - /unlink 24209 (time 1721879156.13 total 48.78 last 498.89) - /unlink 30000 (time 1721879164.82 total 57.47 last 666.36) - /unlink 37267 (time 1721879174.82 total 67.47 last 726.62) - /unlink 40000 (time 1721879178.96 total 71.62 last 658.95) - /unlink 45802 (time 1721879188.97 total 81.62 last 580.10) - /unlink 50000 (time 1721879195.18 total 87.84 last 675.33) - /unlink 56215 (time 1721879205.18 total 97.84 last 621.49) - /unlink 60000 (time 1721879209.91 total 102.57 last 800.00) total: 64768 /unlink in 108.20 seconds: 598.61 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4461 (time 1721879247.71 total 10.00 last 446.08) - open/close 8628 (time 1721879257.71 total 20.00 last 416.64) - open/close 10000 (time 1721879260.34 total 22.63 last 521.24) - open/close 15371 (time 1721879270.34 total 32.63 last 537.08) - open/close 20000 (time 1721879279.77 total 42.06 last 491.00) - open/close 25262 (time 1721879289.77 total 52.06 last 526.17) - open/close 30000 (time 1721879299.54 total 61.83 last 484.99) - open/close 35128 (time 1721879309.54 total 71.83 last 512.74) - open/close 40000 (time 1721879318.99 total 81.29 last 515.27) - open/close 45129 (time 1721879329.00 total 91.29 last 512.80) - open/close 50000 (time 1721879338.68 total 100.97 last 503.19) - open/close 55036 (time 1721879348.68 total 110.97 last 503.51) - open/close 60000 (time 1721879358.41 total 120.70 last 510.19) total: 64768 open/close in 130.17 seconds: 497.56 ops/second - /unlink 8562 (time 1721879378.74 total 10.00 last 856.13) - /unlink 17152 (time 1721879388.74 total 20.00 last 858.97) - /unlink 20000 (time 1721879392.16 total 23.42 last 833.54) - /unlink 28521 (time 1721879402.16 total 33.42 last 852.05) - /unlink 36968 (time 1721879412.16 total 43.42 last 844.64) - /unlink 40000 (time 1721879415.65 total 46.91 last 867.74) - /unlink 48587 (time 1721879425.65 total 56.91 last 858.65) - /unlink 57271 (time 1721879435.66 total 66.91 last 868.34) - /unlink 60000 (time 1721879438.81 total 70.06 last 866.37) total: 64768 /unlink in 75.37 seconds: 859.37 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5194 (time 1721879478.40 total 10.00 last 519.32) - open/close 10000 (time 1721879487.78 total 19.39 last 511.98) - open/close 15163 (time 1721879497.78 total 29.39 last 516.27) - open/close 20000 (time 1721879507.23 total 38.83 last 512.22) - open/close 25129 (time 1721879517.23 total 48.83 last 512.87) - open/close 30000 (time 1721879526.69 total 58.30 last 514.67) - open/close 35052 (time 1721879536.69 total 68.30 last 505.20) - open/close 40000 (time 1721879546.51 total 78.12 last 503.94) - open/close 45186 (time 1721879556.51 total 88.12 last 518.57) - open/close 50000 (time 1721879566.12 total 97.72 last 501.09) - open/close 55077 (time 1721879576.12 total 107.72 last 507.68) - open/close 60000 (time 1721879585.68 total 117.28 last 514.97) total: 64768 open/close in 126.89 seconds: 510.43 ops/second - /unlink 8608 (time 1721879606.13 total 10.00 last 860.74) - /unlink 16742 (time 1721879616.13 total 20.00 last 813.35) - /unlink 20000 (time 1721879620.89 total 24.76 last 683.93) - /unlink 25416 (time 1721879630.89 total 34.77 last 541.55) - /unlink 30000 (time 1721879636.29 total 40.17 last 848.85) - /unlink 35648 (time 1721879646.29 total 50.17 last 564.78) - /unlink 39308 (time 1721879656.29 total 60.17 last 365.99) - /unlink 47468 (time 1721879666.29 total 70.17 last 815.99) - /unlink 50000 (time 1721879669.06 total 72.93 last 916.16) - /unlink 57656 (time 1721879679.06 total 82.93 last 765.57) - /unlink 60000 (time 1721879683.37 total 87.25 last 543.21) total: 64768 /unlink in 95.46 seconds: 678.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4824 (time 1721879719.18 total 10.00 last 482.30) - open/close 9304 (time 1721879729.18 total 20.00 last 447.98) - open/close 13986 (time 1721879739.18 total 30.00 last 468.15) - open/close 19477 (time 1721879749.18 total 40.00 last 549.07) - open/close 23997 (time 1721879759.19 total 50.01 last 451.93) - open/close 28400 (time 1721879769.19 total 60.01 last 440.28) - open/close 30000 (time 1721879772.62 total 63.44 last 465.70) - open/close 34716 (time 1721879782.62 total 73.44 last 471.57) - open/close 40000 (time 1721879792.57 total 83.39 last 531.10) - open/close 45497 (time 1721879802.57 total 93.39 last 549.61) - open/close 50000 (time 1721879811.12 total 101.94 last 526.84) - open/close 55495 (time 1721879821.12 total 111.94 last 549.49) - open/close 60000 (time 1721879829.12 total 119.94 last 562.91) total: 64768 open/close in 128.14 seconds: 505.45 ops/second - /unlink 9419 (time 1721879848.09 total 10.00 last 941.83) - /unlink 19158 (time 1721879858.09 total 20.00 last 973.81) - /unlink 28662 (time 1721879868.09 total 30.00 last 950.36) - /unlink 36998 (time 1721879878.09 total 40.00 last 833.58) - /unlink 40000 (time 1721879881.82 total 43.74 last 804.16) - /unlink 48172 (time 1721879891.83 total 53.74 last 816.71) - /unlink 50000 (time 1721879894.09 total 56.01 last 807.51) - /unlink 58251 (time 1721879904.09 total 66.01 last 825.06) total: 64768 /unlink in 72.49 seconds: 893.44 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server unloading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2