== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:35:25 (1747809325) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5658 (time 1747809359.07 total 10.00 last 565.77) - open/close 10000 (time 1747809366.77 total 17.70 last 563.69) - open/close 15648 (time 1747809376.77 total 27.70 last 564.76) - open/close 20000 (time 1747809384.42 total 35.35 last 569.24) - open/close 25537 (time 1747809394.42 total 45.35 last 553.65) - open/close 30000 (time 1747809402.56 total 53.49 last 548.43) - open/close 35560 (time 1747809412.56 total 63.49 last 555.95) - open/close 40000 (time 1747809420.37 total 71.30 last 568.45) - open/close 45491 (time 1747809430.37 total 81.30 last 549.05) - open/close 50000 (time 1747809438.60 total 89.53 last 548.00) - open/close 55559 (time 1747809448.60 total 99.53 last 555.83) - open/close 60000 (time 1747809456.81 total 107.74 last 541.12) total: 64768 open/close in 116.48 seconds: 556.06 ops/second - /unlink 9640 (time 1747809476.36 total 10.00 last 963.93) - /unlink 18796 (time 1747809486.36 total 20.00 last 915.59) - /unlink 27985 (time 1747809496.36 total 30.00 last 918.85) - /unlink 30000 (time 1747809498.50 total 32.14 last 943.00) - /unlink 39082 (time 1747809508.50 total 42.14 last 908.16) - /unlink 48240 (time 1747809518.50 total 52.14 last 915.79) - /unlink 57509 (time 1747809528.50 total 62.14 last 926.84) - /unlink 60000 (time 1747809531.37 total 65.01 last 868.02) total: 64768 /unlink in 70.00 seconds: 925.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5291 (time 1747809572.40 total 10.00 last 529.04) - open/close 10000 (time 1747809581.48 total 19.08 last 518.75) - open/close 15368 (time 1747809591.48 total 29.08 last 536.78) - open/close 20000 (time 1747809600.21 total 37.81 last 530.42) - open/close 25229 (time 1747809610.21 total 47.81 last 522.90) - open/close 30000 (time 1747809619.14 total 56.74 last 534.51) - open/close 35321 (time 1747809629.14 total 66.74 last 532.06) - open/close 40000 (time 1747809638.13 total 75.74 last 519.97) - open/close 45105 (time 1747809648.14 total 85.74 last 510.41) - open/close 50000 (time 1747809657.56 total 95.16 last 519.36) - open/close 55153 (time 1747809667.56 total 105.16 last 515.27) - open/close 60000 (time 1747809677.10 total 114.70 last 508.25) total: 64768 open/close in 123.95 seconds: 522.51 ops/second - /unlink 9240 (time 1747809697.20 total 10.00 last 923.99) - /unlink 18141 (time 1747809707.20 total 20.00 last 890.10) - /unlink 27335 (time 1747809717.20 total 30.00 last 919.36) - /unlink 30000 (time 1747809719.97 total 32.77 last 962.39) - /unlink 38907 (time 1747809729.97 total 42.77 last 890.66) - /unlink 47946 (time 1747809739.97 total 52.77 last 903.81) - /unlink 50000 (time 1747809742.36 total 55.15 last 862.04) - /unlink 58886 (time 1747809752.36 total 65.15 last 888.59) total: 64768 /unlink in 71.96 seconds: 900.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4561 (time 1747809798.54 total 10.00 last 456.04) - open/close 8066 (time 1747809808.55 total 20.00 last 350.43) - open/close 10000 (time 1747809814.64 total 26.10 last 317.27) - open/close 13322 (time 1747809824.64 total 36.10 last 332.16) - open/close 16978 (time 1747809834.64 total 46.10 last 365.53) - open/close 20000 (time 1747809843.22 total 54.67 last 352.59) - open/close 23343 (time 1747809853.22 total 64.67 last 334.27) - open/close 26462 (time 1747809863.22 total 74.67 last 311.90) - open/close 29966 (time 1747809873.22 total 84.68 last 350.35) - open/close 34575 (time 1747809883.22 total 94.68 last 460.90) - open/close 38883 (time 1747809893.22 total 104.68 last 430.79) - open/close 40000 (time 1747809895.98 total 107.44 last 403.73) - open/close 43749 (time 1747809905.99 total 117.44 last 374.87) - open/close 47620 (time 1747809915.99 total 127.44 last 387.04) - open/close 50000 (time 1747809921.51 total 132.96 last 431.20) - open/close 53687 (time 1747809931.51 total 142.97 last 368.66) - open/close 57395 (time 1747809941.51 total 152.97 last 370.71) - open/close 60000 (time 1747809949.40 total 160.86 last 329.99) - open/close 63371 (time 1747809959.41 total 170.86 last 337.05) total: 64768 open/close in 174.06 seconds: 372.11 ops/second - /unlink 8370 (time 1747809973.58 total 10.00 last 836.93) - /unlink 16287 (time 1747809983.58 total 20.00 last 791.65) - /unlink 20000 (time 1747809987.63 total 24.05 last 917.34) - /unlink 27262 (time 1747809997.63 total 34.05 last 726.05) - /unlink 30000 (time 1747810001.36 total 37.79 last 732.63) - /unlink 38944 (time 1747810011.37 total 47.79 last 894.31) - /unlink 46568 (time 1747810021.37 total 57.79 last 762.35) - /unlink 50000 (time 1747810025.23 total 61.65 last 888.86) - /unlink 56132 (time 1747810035.23 total 71.65 last 613.17) - /unlink 60000 (time 1747810040.54 total 76.96 last 728.67) total: 64768 /unlink in 83.46 seconds: 776.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3430 (time 1747810073.51 total 10.00 last 342.94) - open/close 6394 (time 1747810083.51 total 20.00 last 296.38) - open/close 10000 (time 1747810092.16 total 28.65 last 416.80) - open/close 14046 (time 1747810102.16 total 38.66 last 404.54) - open/close 17318 (time 1747810112.16 total 48.66 last 327.13) - open/close 20000 (time 1747810120.09 total 56.58 last 338.41) - open/close 23831 (time 1747810130.09 total 66.59 last 383.00) - open/close 27355 (time 1747810140.10 total 76.59 last 352.30) - open/close 30000 (time 1747810147.16 total 83.66 last 374.14) - open/close 33752 (time 1747810157.17 total 93.66 last 375.10) - open/close 36533 (time 1747810167.17 total 103.66 last 278.01) - open/close 39740 (time 1747810177.17 total 113.67 last 320.62) - open/close 44124 (time 1747810187.17 total 123.67 last 438.38) - open/close 47996 (time 1747810197.18 total 133.67 last 387.12) - open/close 50000 (time 1747810202.29 total 138.78 last 392.08) - open/close 54324 (time 1747810212.29 total 148.78 last 432.32) - open/close 58168 (time 1747810222.29 total 158.78 last 384.38) - open/close 60000 (time 1747810227.99 total 164.48 last 321.30) total: 64768 open/close in 174.38 seconds: 371.41 ops/second - /unlink 8038 (time 1747810248.68 total 10.00 last 803.68) - /unlink 10000 (time 1747810251.66 total 12.98 last 658.15) - /unlink 17603 (time 1747810261.66 total 22.98 last 760.28) - /unlink 20000 (time 1747810265.72 total 27.04 last 590.94) - /unlink 25601 (time 1747810275.72 total 37.04 last 560.04) - /unlink 30000 (time 1747810284.46 total 45.78 last 503.55) - /unlink 36121 (time 1747810294.46 total 55.78 last 612.01) - /unlink 40000 (time 1747810300.21 total 61.52 last 675.05) - /unlink 49297 (time 1747810310.21 total 71.52 last 929.66) - /unlink 58909 (time 1747810320.21 total 81.52 last 961.13) total: 64768 /unlink in 87.53 seconds: 739.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4451 (time 1747810359.56 total 10.00 last 444.97) - open/close 8013 (time 1747810369.56 total 20.01 last 356.12) - open/close 10000 (time 1747810376.36 total 26.80 last 292.21) - open/close 14502 (time 1747810386.36 total 36.81 last 450.02) - open/close 18817 (time 1747810396.37 total 46.81 last 431.48) - open/close 20000 (time 1747810398.85 total 49.29 last 476.16) - open/close 23102 (time 1747810408.85 total 59.30 last 310.14) - open/close 26725 (time 1747810418.85 total 69.30 last 362.20) - open/close 30000 (time 1747810428.68 total 79.12 last 333.39) - open/close 33186 (time 1747810438.68 total 89.12 last 318.57) - open/close 37270 (time 1747810448.68 total 99.12 last 408.32) - open/close 40000 (time 1747810456.35 total 106.79 last 356.11) - open/close 44055 (time 1747810466.35 total 116.79 last 405.44) - open/close 47289 (time 1747810476.35 total 126.80 last 323.29) - open/close 50000 (time 1747810485.04 total 135.49 last 311.86) - open/close 52783 (time 1747810495.05 total 145.49 last 278.27) - open/close 55593 (time 1747810505.05 total 155.49 last 281.00) - open/close 58836 (time 1747810515.05 total 165.49 last 324.30) - open/close 60000 (time 1747810519.14 total 169.59 last 284.15) - open/close 63127 (time 1747810529.14 total 179.59 last 312.64) total: 64768 open/close in 185.60 seconds: 348.96 ops/second - /unlink 5208 (time 1747810546.33 total 10.00 last 520.77) - /unlink 10000 (time 1747810555.04 total 18.71 last 550.37) - /unlink 16189 (time 1747810565.04 total 28.71 last 618.86) - /unlink 20000 (time 1747810571.91 total 35.58 last 554.75) - /unlink 25430 (time 1747810581.91 total 45.58 last 542.94) - /unlink 30000 (time 1747810588.25 total 51.92 last 721.05) - /unlink 35355 (time 1747810598.25 total 61.92 last 535.46) - /unlink 40000 (time 1747810605.82 total 69.49 last 613.68) - /unlink 46228 (time 1747810615.82 total 79.49 last 622.74) - /unlink 50000 (time 1747810621.32 total 84.98 last 686.21) - /unlink 58493 (time 1747810631.32 total 94.98 last 849.28) total: 64768 /unlink in 102.24 seconds: 633.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2