== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:18:42 (1741691922) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg455-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4768 (time 1741691955.82 total 10.00 last 476.79) - open/close 9075 (time 1741691965.82 total 20.00 last 430.57) - open/close 10000 (time 1741691967.85 total 22.03 last 456.07) - open/close 14822 (time 1741691977.85 total 32.03 last 482.18) - open/close 20000 (time 1741691986.62 total 40.80 last 590.67) - open/close 25845 (time 1741691996.62 total 50.80 last 584.40) - open/close 30000 (time 1741692004.33 total 58.51 last 538.98) - open/close 35910 (time 1741692014.33 total 68.51 last 590.91) - open/close 40000 (time 1741692021.93 total 76.12 last 537.79) - open/close 45386 (time 1741692031.94 total 86.12 last 538.51) - open/close 50000 (time 1741692041.00 total 95.18 last 509.19) - open/close 55204 (time 1741692051.00 total 105.18 last 520.36) - open/close 60000 (time 1741692060.13 total 114.32 last 524.88) total: 64768 open/close in 124.16 seconds: 521.67 ops/second - /unlink 8858 (time 1741692081.06 total 10.00 last 885.76) - /unlink 17939 (time 1741692091.06 total 20.00 last 908.02) - /unlink 20000 (time 1741692093.29 total 22.23 last 925.46) - /unlink 29093 (time 1741692103.29 total 32.23 last 909.22) - /unlink 38513 (time 1741692113.29 total 42.23 last 941.87) - /unlink 47811 (time 1741692123.29 total 52.23 last 929.70) - /unlink 50000 (time 1741692125.52 total 54.47 last 979.60) - /unlink 59518 (time 1741692135.52 total 64.47 last 951.76) total: 64768 /unlink in 69.93 seconds: 926.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5172 (time 1741692178.59 total 10.00 last 517.16) - open/close 10000 (time 1741692187.89 total 19.30 last 519.40) - open/close 15227 (time 1741692197.89 total 29.30 last 522.70) - open/close 20000 (time 1741692207.13 total 38.54 last 516.44) - open/close 25294 (time 1741692217.13 total 48.54 last 529.35) - open/close 30000 (time 1741692226.06 total 57.47 last 526.98) - open/close 35187 (time 1741692236.07 total 67.47 last 518.58) - open/close 40000 (time 1741692245.43 total 76.83 last 514.23) - open/close 45192 (time 1741692255.43 total 86.83 last 519.15) - open/close 50000 (time 1741692264.80 total 96.20 last 513.03) - open/close 55322 (time 1741692274.80 total 106.21 last 532.13) - open/close 60000 (time 1741692283.76 total 115.17 last 521.97) total: 64768 open/close in 124.28 seconds: 521.15 ops/second - /unlink 9245 (time 1741692303.70 total 10.00 last 924.42) - /unlink 18453 (time 1741692313.70 total 20.00 last 920.72) - /unlink 27343 (time 1741692323.70 total 30.00 last 888.91) - /unlink 30000 (time 1741692326.57 total 32.87 last 927.35) - /unlink 39072 (time 1741692336.57 total 42.87 last 907.19) - /unlink 48199 (time 1741692346.57 total 52.87 last 912.62) - /unlink 50000 (time 1741692348.58 total 54.89 last 893.04) - /unlink 59590 (time 1741692358.58 total 64.89 last 958.93) total: 64768 /unlink in 70.25 seconds: 921.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5430 (time 1741692403.87 total 10.00 last 542.91) - open/close 10000 (time 1741692412.18 total 18.32 last 549.67) - open/close 15541 (time 1741692422.18 total 28.32 last 554.07) - open/close 20000 (time 1741692430.38 total 36.51 last 543.92) - open/close 25421 (time 1741692440.38 total 46.51 last 542.08) - open/close 30000 (time 1741692448.77 total 54.90 last 545.92) - open/close 35289 (time 1741692458.77 total 64.90 last 528.81) - open/close 40000 (time 1741692467.84 total 73.98 last 519.18) - open/close 44319 (time 1741692477.84 total 83.98 last 431.86) - open/close 49265 (time 1741692487.84 total 93.98 last 494.57) - open/close 54295 (time 1741692497.85 total 103.98 last 502.91) - open/close 59465 (time 1741692507.85 total 113.98 last 516.92) - open/close 64540 (time 1741692517.85 total 123.98 last 507.38) total: 64768 open/close in 124.65 seconds: 519.59 ops/second - /unlink 9387 (time 1741692529.40 total 10.00 last 938.69) - /unlink 18571 (time 1741692539.40 total 20.00 last 918.38) - /unlink 27832 (time 1741692549.40 total 30.00 last 926.08) - /unlink 30000 (time 1741692551.67 total 32.26 last 957.65) - /unlink 39302 (time 1741692561.67 total 42.26 last 930.17) - /unlink 48636 (time 1741692571.67 total 52.26 last 933.39) - /unlink 58009 (time 1741692581.67 total 62.27 last 937.22) - /unlink 60000 (time 1741692584.16 total 64.76 last 797.59) total: 64768 /unlink in 69.88 seconds: 926.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4896 (time 1741692629.51 total 10.00 last 489.55) - open/close 10000 (time 1741692639.43 total 19.92 last 514.66) - open/close 14595 (time 1741692649.43 total 29.92 last 459.39) - open/close 19268 (time 1741692659.43 total 39.92 last 467.23) - open/close 23432 (time 1741692669.43 total 49.92 last 416.32) - open/close 27910 (time 1741692679.43 total 59.93 last 447.74) - open/close 30000 (time 1741692684.34 total 64.83 last 426.00) - open/close 34134 (time 1741692694.34 total 74.83 last 413.36) - open/close 38098 (time 1741692704.34 total 84.83 last 396.32) - open/close 40000 (time 1741692709.26 total 89.76 last 386.45) - open/close 44127 (time 1741692719.27 total 99.76 last 412.63) - open/close 48318 (time 1741692729.27 total 109.76 last 419.09) - open/close 50000 (time 1741692732.80 total 113.30 last 475.23) - open/close 54731 (time 1741692742.81 total 123.30 last 473.05) - open/close 59822 (time 1741692752.81 total 133.30 last 509.08) total: 64768 open/close in 142.67 seconds: 453.96 ops/second - /unlink 9262 (time 1741692773.01 total 10.00 last 926.11) - /unlink 17986 (time 1741692783.01 total 20.00 last 872.33) - /unlink 20000 (time 1741692785.46 total 22.46 last 819.81) - /unlink 26408 (time 1741692795.46 total 32.46 last 640.78) - /unlink 30000 (time 1741692801.27 total 38.26 last 618.88) - /unlink 36737 (time 1741692811.27 total 48.26 last 673.63) - /unlink 40000 (time 1741692816.62 total 53.62 last 609.23) - /unlink 46896 (time 1741692826.62 total 63.62 last 689.59) - /unlink 50000 (time 1741692831.78 total 68.78 last 601.54) - /unlink 55980 (time 1741692841.79 total 78.78 last 597.96) - /unlink 60000 (time 1741692849.44 total 86.43 last 525.47) total: 64768 /unlink in 95.64 seconds: 677.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3489 (time 1741692884.58 total 10.00 last 348.83) - open/close 6327 (time 1741692894.58 total 20.00 last 283.79) - open/close 9553 (time 1741692904.58 total 30.01 last 322.52) - open/close 12942 (time 1741692914.59 total 40.01 last 338.85) - open/close 15955 (time 1741692924.59 total 50.01 last 301.28) - open/close 18867 (time 1741692934.59 total 60.01 last 291.15) - open/close 20000 (time 1741692938.33 total 63.75 last 303.10) - open/close 22905 (time 1741692948.33 total 73.75 last 290.49) - open/close 25696 (time 1741692958.33 total 83.75 last 279.10) - open/close 28422 (time 1741692968.33 total 93.75 last 272.54) - open/close 30000 (time 1741692974.09 total 99.51 last 273.94) - open/close 32644 (time 1741692984.09 total 109.51 last 264.40) - open/close 35339 (time 1741692994.09 total 119.51 last 269.47) - open/close 38430 (time 1741693004.09 total 129.51 last 309.09) - open/close 40000 (time 1741693008.85 total 134.27 last 330.12) - open/close 42719 (time 1741693018.85 total 144.27 last 271.90) - open/close 45295 (time 1741693028.85 total 154.27 last 257.55) - open/close 47953 (time 1741693038.85 total 164.27 last 265.71) - open/close 50000 (time 1741693046.94 total 172.37 last 252.96) - open/close 52645 (time 1741693056.94 total 182.37 last 264.49) - open/close 55657 (time 1741693066.95 total 192.37 last 301.15) - open/close 58421 (time 1741693076.95 total 202.37 last 276.37) - open/close 60000 (time 1741693083.04 total 208.46 last 259.07) - open/close 62521 (time 1741693093.04 total 218.46 last 252.08) total: 64768 open/close in 226.96 seconds: 285.37 ops/second - /unlink 4837 (time 1741693112.85 total 10.00 last 483.67) - /unlink 9796 (time 1741693122.85 total 20.00 last 495.87) - /unlink 16086 (time 1741693132.85 total 30.00 last 628.97) - /unlink 20000 (time 1741693137.45 total 34.60 last 851.71) - /unlink 25424 (time 1741693147.45 total 44.60 last 542.35) - /unlink 30000 (time 1741693156.37 total 53.52 last 513.07) - /unlink 34965 (time 1741693166.37 total 63.52 last 496.46) - /unlink 40000 (time 1741693176.11 total 73.26 last 516.77) - /unlink 45344 (time 1741693186.11 total 83.26 last 534.34) - /unlink 50000 (time 1741693195.19 total 92.34 last 512.71) - /unlink 54922 (time 1741693205.20 total 102.34 last 492.17) - /unlink 59772 (time 1741693215.20 total 112.35 last 484.94) total: 64768 /unlink in 121.03 seconds: 535.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2