== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:16:06 (1743527766) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) oleg409-server: oleg409-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg409-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5338 (time 1743527801.21 total 10.00 last 533.72) - open/close 9665 (time 1743527811.21 total 20.00 last 432.67) - open/close 13418 (time 1743527821.21 total 30.00 last 375.27) - open/close 16894 (time 1743527831.21 total 40.01 last 347.49) - open/close 20000 (time 1743527840.52 total 49.31 last 333.77) - open/close 24046 (time 1743527850.52 total 59.31 last 404.59) - open/close 28255 (time 1743527860.52 total 69.31 last 420.88) - open/close 30000 (time 1743527864.02 total 72.82 last 497.64) - open/close 34735 (time 1743527874.02 total 82.82 last 473.50) - open/close 40000 (time 1743527883.90 total 92.69 last 533.40) - open/close 44281 (time 1743527893.90 total 102.69 last 428.06) - open/close 48686 (time 1743527903.90 total 112.69 last 440.48) - open/close 50000 (time 1743527906.60 total 115.39 last 486.05) - open/close 53717 (time 1743527916.60 total 125.40 last 371.59) - open/close 57708 (time 1743527926.60 total 135.40 last 399.09) - open/close 60000 (time 1743527933.19 total 141.98 last 348.14) - open/close 63715 (time 1743527943.19 total 151.98 last 371.45) total: 64768 open/close in 154.69 seconds: 418.70 ops/second - /unlink 5772 (time 1743527956.82 total 10.00 last 577.17) - /unlink 10000 (time 1743527963.26 total 16.44 last 656.51) - /unlink 18752 (time 1743527973.26 total 26.44 last 874.94) - /unlink 27670 (time 1743527983.26 total 36.44 last 891.78) - /unlink 30000 (time 1743527986.11 total 39.29 last 818.69) - /unlink 36565 (time 1743527996.11 total 49.29 last 656.42) - /unlink 40000 (time 1743528001.06 total 54.24 last 693.57) - /unlink 47137 (time 1743528011.06 total 64.24 last 713.65) - /unlink 50000 (time 1743528015.24 total 68.42 last 685.57) - /unlink 57734 (time 1743528025.24 total 78.42 last 773.31) - /unlink 60000 (time 1743528028.60 total 81.78 last 674.92) total: 64768 /unlink in 89.23 seconds: 725.87 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3448 (time 1743528065.14 total 10.00 last 344.76) - open/close 7208 (time 1743528075.14 total 20.00 last 375.98) - open/close 10000 (time 1743528083.23 total 28.09 last 345.10) - open/close 13604 (time 1743528093.23 total 38.09 last 360.32) - open/close 17770 (time 1743528103.24 total 48.10 last 416.57) - open/close 20000 (time 1743528107.77 total 52.63 last 491.94) - open/close 24678 (time 1743528117.77 total 62.63 last 467.74) - open/close 30000 (time 1743528127.40 total 72.26 last 552.77) - open/close 34357 (time 1743528137.40 total 82.26 last 435.69) - open/close 38846 (time 1743528147.40 total 92.26 last 448.89) - open/close 40000 (time 1743528149.65 total 94.51 last 512.81) - open/close 45280 (time 1743528159.65 total 104.51 last 527.99) - open/close 50000 (time 1743528168.23 total 113.10 last 549.68) - open/close 55728 (time 1743528178.24 total 123.10 last 572.78) - open/close 60000 (time 1743528186.54 total 131.40 last 514.59) total: 64768 open/close in 140.80 seconds: 460.00 ops/second - /unlink 9965 (time 1743528206.70 total 10.00 last 996.48) - /unlink 19440 (time 1743528216.70 total 20.00 last 947.45) - /unlink 29204 (time 1743528226.71 total 30.00 last 976.37) - /unlink 38602 (time 1743528236.71 total 40.00 last 939.76) - /unlink 47104 (time 1743528246.71 total 50.00 last 850.14) - /unlink 50000 (time 1743528250.03 total 53.33 last 870.95) - /unlink 58720 (time 1743528260.03 total 63.33 last 871.90) total: 64768 /unlink in 70.24 seconds: 922.15 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4265 (time 1743528305.55 total 10.00 last 426.46) - open/close 9399 (time 1743528315.55 total 20.00 last 513.34) - open/close 14876 (time 1743528325.55 total 30.00 last 547.68) - open/close 20000 (time 1743528334.85 total 39.30 last 550.94) - open/close 24700 (time 1743528344.85 total 49.30 last 469.95) - open/close 29582 (time 1743528354.85 total 59.31 last 488.16) - open/close 34383 (time 1743528364.85 total 69.31 last 480.01) - open/close 39203 (time 1743528374.85 total 79.31 last 481.95) - open/close 44249 (time 1743528384.85 total 89.31 last 504.52) - open/close 49708 (time 1743528394.86 total 99.31 last 545.83) - open/close 55072 (time 1743528404.86 total 109.31 last 536.33) - open/close 60000 (time 1743528414.34 total 118.79 last 519.76) total: 64768 open/close in 127.46 seconds: 508.13 ops/second - /unlink 10000 (time 1743528433.51 total 9.72 last 1029.33) - /unlink 19534 (time 1743528443.51 total 19.72 last 953.32) - /unlink 29144 (time 1743528453.51 total 29.72 last 960.92) - /unlink 38340 (time 1743528463.51 total 39.72 last 919.51) - /unlink 47919 (time 1743528473.51 total 49.72 last 957.82) - /unlink 50000 (time 1743528475.61 total 51.81 last 992.79) - /unlink 60000 (time 1743528485.51 total 61.72 last 1009.80) total: 64768 /unlink in 66.82 seconds: 969.28 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4781 (time 1743528531.29 total 10.00 last 478.05) - open/close 9443 (time 1743528541.29 total 20.00 last 466.19) - open/close 14238 (time 1743528551.30 total 30.00 last 479.45) - open/close 19306 (time 1743528561.30 total 40.00 last 506.76) - open/close 24439 (time 1743528571.30 total 50.00 last 513.29) - open/close 29565 (time 1743528581.30 total 60.00 last 512.59) - open/close 34512 (time 1743528591.30 total 70.00 last 494.70) - open/close 39443 (time 1743528601.30 total 80.00 last 493.06) - open/close 44123 (time 1743528611.30 total 90.01 last 467.92) - open/close 49234 (time 1743528621.30 total 100.01 last 511.01) - open/close 54989 (time 1743528631.30 total 110.01 last 575.44) - open/close 60000 (time 1743528641.28 total 119.99 last 501.99) total: 64768 open/close in 129.97 seconds: 498.31 ops/second - /unlink 8931 (time 1743528662.14 total 10.00 last 893.03) - /unlink 17593 (time 1743528672.14 total 20.00 last 866.11) - /unlink 20000 (time 1743528674.93 total 22.79 last 862.11) - /unlink 28112 (time 1743528684.93 total 32.79 last 811.19) - /unlink 30000 (time 1743528687.36 total 35.22 last 776.65) - /unlink 38672 (time 1743528697.36 total 45.23 last 867.12) - /unlink 47555 (time 1743528707.36 total 55.23 last 888.28) - /unlink 50000 (time 1743528710.27 total 58.14 last 840.11) - /unlink 58997 (time 1743528720.28 total 68.14 last 899.68) total: 64768 /unlink in 74.30 seconds: 871.76 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5134 (time 1743528761.18 total 10.00 last 513.35) - open/close 10000 (time 1743528770.19 total 19.01 last 540.28) - open/close 15090 (time 1743528780.19 total 29.01 last 508.96) - open/close 20000 (time 1743528789.72 total 38.54 last 515.35) - open/close 25227 (time 1743528799.72 total 48.54 last 522.63) - open/close 30000 (time 1743528809.51 total 58.33 last 487.28) - open/close 34981 (time 1743528819.51 total 68.33 last 498.03) - open/close 38231 (time 1743528829.52 total 78.33 last 324.98) - open/close 40000 (time 1743528833.88 total 82.70 last 405.16) - open/close 44410 (time 1743528843.88 total 92.70 last 440.91) - open/close 48536 (time 1743528853.88 total 102.70 last 412.59) - open/close 50000 (time 1743528857.64 total 106.46 last 389.55) - open/close 53371 (time 1743528867.64 total 116.46 last 337.02) - open/close 56739 (time 1743528877.65 total 126.47 last 336.70) - open/close 60000 (time 1743528885.10 total 133.92 last 437.64) - open/close 64017 (time 1743528895.10 total 143.92 last 401.68) total: 64768 open/close in 145.48 seconds: 445.21 ops/second - /unlink 7208 (time 1743528907.53 total 10.00 last 720.75) - /unlink 10000 (time 1743528911.40 total 13.87 last 720.79) - /unlink 18835 (time 1743528921.40 total 23.88 last 883.42) - /unlink 28132 (time 1743528931.40 total 33.88 last 929.66) - /unlink 37261 (time 1743528941.40 total 43.88 last 912.86) - /unlink 40000 (time 1743528944.33 total 46.80 last 936.73) - /unlink 49302 (time 1743528954.33 total 56.80 last 930.14) - /unlink 58648 (time 1743528964.33 total 66.80 last 934.58) total: 64768 /unlink in 73.32 seconds: 883.32 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server unloading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2