== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:50:34 (1748469034) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg416-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5520 (time 1748469068.16 total 10.00 last 551.97) - open/close 10000 (time 1748469076.04 total 17.88 last 568.77) - open/close 15781 (time 1748469086.04 total 27.88 last 578.04) - open/close 20000 (time 1748469093.48 total 35.32 last 566.85) - open/close 25609 (time 1748469103.48 total 45.32 last 560.81) - open/close 30000 (time 1748469111.29 total 53.13 last 562.43) - open/close 35687 (time 1748469121.29 total 63.13 last 568.64) - open/close 40000 (time 1748469129.00 total 70.84 last 559.69) - open/close 45392 (time 1748469139.00 total 80.84 last 539.16) - open/close 50000 (time 1748469147.63 total 89.47 last 534.07) - open/close 55335 (time 1748469157.63 total 99.47 last 533.41) - open/close 60000 (time 1748469166.25 total 108.09 last 541.22) total: 64768 open/close in 116.81 seconds: 554.49 ops/second - /unlink 9303 (time 1748469185.79 total 10.00 last 930.28) - /unlink 18504 (time 1748469195.80 total 20.00 last 920.03) - /unlink 27605 (time 1748469205.80 total 30.00 last 910.06) - /unlink 30000 (time 1748469208.29 total 32.49 last 960.55) - /unlink 38959 (time 1748469218.29 total 42.50 last 895.83) - /unlink 47963 (time 1748469228.29 total 52.50 last 900.32) - /unlink 50000 (time 1748469230.65 total 54.85 last 863.82) - /unlink 59183 (time 1748469240.65 total 64.85 last 918.27) total: 64768 /unlink in 70.90 seconds: 913.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5609 (time 1748469282.02 total 10.00 last 560.84) - open/close 10000 (time 1748469290.02 total 17.99 last 549.41) - open/close 15308 (time 1748469300.02 total 27.99 last 530.80) - open/close 20000 (time 1748469308.71 total 36.69 last 539.46) - open/close 25399 (time 1748469318.71 total 46.69 last 539.84) - open/close 30000 (time 1748469327.68 total 55.66 last 513.26) - open/close 35239 (time 1748469337.68 total 65.66 last 523.87) - open/close 40000 (time 1748469346.65 total 74.62 last 531.03) - open/close 45285 (time 1748469356.65 total 84.62 last 528.40) - open/close 50000 (time 1748469365.72 total 93.70 last 519.62) - open/close 55369 (time 1748469375.72 total 103.70 last 536.90) - open/close 60000 (time 1748469384.61 total 112.58 last 521.13) total: 64768 open/close in 121.56 seconds: 532.79 ops/second - /unlink 9150 (time 1748469404.40 total 10.00 last 914.91) - /unlink 18399 (time 1748469414.40 total 20.00 last 924.90) - /unlink 27669 (time 1748469424.40 total 30.00 last 926.98) - /unlink 30000 (time 1748469427.00 total 32.60 last 896.59) - /unlink 39281 (time 1748469437.01 total 42.60 last 928.07) - /unlink 48487 (time 1748469447.01 total 52.60 last 920.53) - /unlink 57699 (time 1748469457.01 total 62.60 last 921.17) - /unlink 60000 (time 1748469459.36 total 64.96 last 976.64) total: 64768 /unlink in 69.98 seconds: 925.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5799 (time 1748469502.50 total 10.00 last 579.83) - open/close 10000 (time 1748469509.78 total 17.28 last 577.26) - open/close 15646 (time 1748469519.78 total 27.28 last 564.58) - open/close 20000 (time 1748469527.47 total 34.97 last 565.99) - open/close 25677 (time 1748469537.47 total 44.97 last 567.65) - open/close 30000 (time 1748469545.26 total 52.76 last 554.89) - open/close 35492 (time 1748469555.26 total 62.76 last 549.13) - open/close 40000 (time 1748469563.31 total 70.81 last 560.21) - open/close 45475 (time 1748469573.31 total 80.81 last 547.48) - open/close 50000 (time 1748469582.33 total 89.84 last 501.41) - open/close 55290 (time 1748469592.33 total 99.84 last 528.99) - open/close 60000 (time 1748469601.39 total 108.89 last 520.01) total: 64768 open/close in 117.66 seconds: 550.47 ops/second - /unlink 8948 (time 1748469620.98 total 10.00 last 894.63) - /unlink 17585 (time 1748469630.99 total 20.00 last 863.57) - /unlink 20000 (time 1748469633.53 total 22.54 last 950.35) - /unlink 29041 (time 1748469643.53 total 32.54 last 904.09) - /unlink 37440 (time 1748469653.53 total 42.55 last 839.87) - /unlink 40000 (time 1748469656.35 total 45.37 last 905.76) - /unlink 49332 (time 1748469666.35 total 55.37 last 933.18) - /unlink 58584 (time 1748469676.35 total 65.37 last 925.20) total: 64768 /unlink in 71.97 seconds: 899.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4355 (time 1748469717.65 total 10.00 last 435.41) - open/close 8622 (time 1748469727.66 total 20.00 last 426.63) - open/close 10000 (time 1748469731.18 total 23.53 last 390.86) - open/close 14289 (time 1748469741.18 total 33.53 last 428.89) - open/close 18547 (time 1748469751.18 total 43.53 last 425.78) - open/close 20000 (time 1748469754.77 total 47.12 last 404.51) - open/close 23178 (time 1748469764.78 total 57.13 last 317.71) - open/close 27212 (time 1748469774.78 total 67.13 last 403.39) - open/close 30000 (time 1748469784.09 total 76.43 last 299.50) - open/close 33213 (time 1748469794.09 total 86.44 last 321.23) - open/close 36089 (time 1748469804.09 total 96.44 last 287.58) - open/close 39337 (time 1748469814.09 total 106.44 last 324.77) - open/close 42313 (time 1748469824.13 total 116.47 last 296.52) - open/close 45209 (time 1748469834.13 total 126.48 last 289.56) - open/close 48271 (time 1748469844.13 total 136.48 last 306.09) - open/close 50000 (time 1748469850.82 total 143.17 last 258.39) - open/close 52713 (time 1748469860.82 total 153.17 last 271.28) - open/close 55517 (time 1748469870.82 total 163.17 last 280.39) - open/close 59244 (time 1748469880.82 total 173.17 last 372.69) - open/close 64359 (time 1748469890.83 total 183.17 last 511.45) total: 64768 open/close in 183.98 seconds: 352.04 ops/second - /unlink 9341 (time 1748469902.42 total 10.00 last 934.04) - /unlink 18545 (time 1748469912.42 total 20.00 last 920.37) - /unlink 25540 (time 1748469922.42 total 30.00 last 699.47) - /unlink 30000 (time 1748469931.01 total 38.59 last 519.53) - /unlink 35012 (time 1748469941.01 total 48.59 last 501.13) - /unlink 39945 (time 1748469951.01 total 58.59 last 493.28) - /unlink 45349 (time 1748469961.01 total 68.59 last 540.32) - /unlink 50000 (time 1748469969.84 total 77.42 last 526.92) - /unlink 55342 (time 1748469979.84 total 87.42 last 534.09) - /unlink 60000 (time 1748469988.41 total 95.99 last 543.41) total: 64768 /unlink in 104.83 seconds: 617.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3278 (time 1748470023.26 total 10.00 last 327.78) - open/close 6639 (time 1748470033.26 total 20.00 last 336.04) - open/close 10000 (time 1748470042.37 total 29.12 last 368.81) - open/close 12793 (time 1748470052.37 total 39.12 last 279.20) - open/close 15457 (time 1748470062.38 total 49.12 last 266.31) - open/close 18215 (time 1748470072.38 total 59.13 last 275.73) - open/close 20000 (time 1748470078.76 total 65.50 last 279.83) - open/close 23026 (time 1748470088.76 total 75.51 last 302.51) - open/close 25718 (time 1748470098.76 total 85.51 last 269.18) - open/close 28392 (time 1748470108.77 total 95.51 last 267.29) - open/close 30000 (time 1748470114.61 total 101.35 last 275.44) - open/close 32638 (time 1748470124.61 total 111.35 last 263.70) - open/close 35359 (time 1748470134.61 total 121.36 last 272.03) - open/close 39130 (time 1748470144.61 total 131.36 last 377.01) - open/close 40000 (time 1748470148.02 total 134.76 last 255.57) - open/close 42726 (time 1748470158.02 total 144.77 last 272.53) - open/close 45530 (time 1748470168.02 total 154.77 last 280.31) - open/close 48292 (time 1748470178.02 total 164.77 last 276.19) - open/close 50000 (time 1748470183.94 total 170.68 last 288.77) - open/close 53236 (time 1748470193.94 total 180.69 last 323.53) - open/close 55957 (time 1748470203.94 total 190.69 last 272.01) - open/close 58496 (time 1748470213.95 total 200.69 last 253.82) - open/close 60000 (time 1748470219.85 total 206.59 last 254.81) - open/close 62634 (time 1748470229.85 total 216.60 last 263.35) total: 64768 open/close in 224.21 seconds: 288.87 ops/second - /unlink 6691 (time 1748470248.74 total 10.00 last 669.09) - /unlink 10000 (time 1748470252.35 total 13.61 last 917.33) - /unlink 16933 (time 1748470262.35 total 23.61 last 693.28) - /unlink 20000 (time 1748470265.93 total 27.19 last 856.78) - /unlink 27744 (time 1748470275.93 total 37.19 last 774.40) - /unlink 30000 (time 1748470278.62 total 39.87 last 839.96) - /unlink 35328 (time 1748470288.62 total 49.87 last 532.77) - /unlink 40000 (time 1748470297.22 total 58.48 last 542.79) - /unlink 45547 (time 1748470307.23 total 68.48 last 554.63) - /unlink 50000 (time 1748470315.84 total 77.09 last 517.18) - /unlink 55093 (time 1748470325.84 total 87.09 last 509.29) - /unlink 60000 (time 1748470333.46 total 94.72 last 643.27) total: 64768 /unlink in 102.08 seconds: 634.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2