== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:57:02 (1744019822) Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:-f) oleg121-server: oleg121-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg121-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg121-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4374 (time 1744019857.42 total 10.00 last 437.33) - open/close 8886 (time 1744019867.42 total 20.00 last 451.19) - open/close 10000 (time 1744019869.99 total 22.57 last 432.96) - open/close 14517 (time 1744019879.99 total 32.58 last 451.67) - open/close 18729 (time 1744019889.99 total 42.58 last 421.17) - open/close 20000 (time 1744019892.74 total 45.33 last 461.57) - open/close 24899 (time 1744019902.75 total 55.33 last 489.83) - open/close 30000 (time 1744019911.60 total 64.18 last 576.30) - open/close 35838 (time 1744019921.60 total 74.18 last 583.80) - open/close 40000 (time 1744019929.54 total 82.13 last 523.98) - open/close 45364 (time 1744019939.54 total 92.13 last 536.32) - open/close 50000 (time 1744019948.31 total 100.89 last 528.94) - open/close 55389 (time 1744019958.31 total 110.89 last 538.88) - open/close 60000 (time 1744019966.63 total 119.22 last 553.83) total: 64768 open/close in 128.16 seconds: 505.39 ops/second - /unlink 8848 (time 1744019986.36 total 10.00 last 884.68) - /unlink 17728 (time 1744019996.36 total 20.00 last 887.92) - /unlink 20000 (time 1744019999.02 total 22.66 last 853.96) - /unlink 28304 (time 1744020009.02 total 32.66 last 830.33) - /unlink 30000 (time 1744020011.09 total 34.73 last 820.10) - /unlink 38129 (time 1744020021.09 total 44.73 last 812.88) - /unlink 40000 (time 1744020023.30 total 46.94 last 847.78) - /unlink 48550 (time 1744020033.30 total 56.94 last 854.92) - /unlink 55886 (time 1744020043.30 total 66.94 last 733.55) - /unlink 60000 (time 1744020048.24 total 71.88 last 832.86) total: 64768 /unlink in 78.00 seconds: 830.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4365 (time 1744020085.71 total 10.00 last 436.47) - open/close 7800 (time 1744020095.71 total 20.00 last 343.49) - open/close 10000 (time 1744020101.33 total 25.62 last 391.43) - open/close 13712 (time 1744020111.34 total 35.62 last 371.15) - open/close 18124 (time 1744020121.34 total 45.62 last 441.16) - open/close 20000 (time 1744020125.47 total 49.76 last 453.79) - open/close 24712 (time 1744020135.47 total 59.76 last 471.18) - open/close 29104 (time 1744020145.47 total 69.76 last 439.13) - open/close 33854 (time 1744020155.47 total 79.76 last 474.92) - open/close 38725 (time 1744020165.48 total 89.76 last 487.06) - open/close 40000 (time 1744020168.12 total 92.41 last 481.80) - open/close 44998 (time 1744020178.12 total 102.41 last 499.76) - open/close 49956 (time 1744020188.12 total 112.41 last 495.80) - open/close 54901 (time 1744020198.12 total 122.41 last 494.47) - open/close 59475 (time 1744020208.12 total 132.41 last 457.33) - open/close 63972 (time 1744020218.13 total 142.41 last 449.66) total: 64768 open/close in 144.18 seconds: 449.22 ops/second - /unlink 8102 (time 1744020230.76 total 10.00 last 810.16) - /unlink 10000 (time 1744020232.95 total 12.19 last 866.24) - /unlink 17770 (time 1744020242.95 total 22.19 last 776.94) - /unlink 20000 (time 1744020245.79 total 25.03 last 785.14) - /unlink 28177 (time 1744020255.79 total 35.03 last 817.69) - /unlink 30000 (time 1744020257.86 total 37.10 last 881.88) - /unlink 38761 (time 1744020267.86 total 47.10 last 876.10) - /unlink 47411 (time 1744020277.86 total 57.10 last 864.93) - /unlink 50000 (time 1744020280.88 total 60.12 last 856.16) - /unlink 58624 (time 1744020290.88 total 70.12 last 862.39) total: 64768 /unlink in 76.75 seconds: 843.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5559 (time 1744020331.05 total 10.00 last 555.87) - open/close 10000 (time 1744020339.09 total 18.04 last 552.16) - open/close 15516 (time 1744020349.09 total 28.04 last 551.55) - open/close 20000 (time 1744020357.66 total 36.61 last 523.33) - open/close 25352 (time 1744020367.66 total 46.61 last 535.12) - open/close 30000 (time 1744020376.47 total 55.42 last 527.64) - open/close 35391 (time 1744020386.47 total 65.42 last 539.05) - open/close 40000 (time 1744020395.22 total 74.17 last 526.83) - open/close 44987 (time 1744020405.22 total 84.17 last 498.65) - open/close 50000 (time 1744020414.84 total 93.79 last 521.07) - open/close 55361 (time 1744020424.84 total 103.79 last 536.08) - open/close 60000 (time 1744020433.70 total 112.66 last 523.51) total: 64768 open/close in 121.49 seconds: 533.09 ops/second - /unlink 9229 (time 1744020453.36 total 10.00 last 922.77) - /unlink 18421 (time 1744020463.36 total 20.00 last 919.17) - /unlink 27500 (time 1744020473.36 total 30.00 last 907.83) - /unlink 30000 (time 1744020476.13 total 32.78 last 900.70) - /unlink 38416 (time 1744020486.13 total 42.78 last 841.59) - /unlink 47577 (time 1744020496.13 total 52.78 last 916.04) - /unlink 50000 (time 1744020498.65 total 55.30 last 961.27) - /unlink 59197 (time 1744020508.65 total 65.30 last 919.69) total: 64768 /unlink in 71.13 seconds: 910.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5164 (time 1744020551.64 total 10.00 last 516.36) - open/close 10000 (time 1744020560.84 total 19.20 last 525.68) - open/close 15288 (time 1744020570.84 total 29.20 last 528.74) - open/close 20000 (time 1744020579.87 total 38.23 last 521.69) - open/close 25308 (time 1744020589.87 total 48.24 last 530.72) - open/close 30000 (time 1744020598.79 total 57.16 last 525.93) - open/close 34600 (time 1744020608.80 total 67.16 last 459.92) - open/close 39195 (time 1744020618.80 total 77.16 last 459.46) - open/close 44281 (time 1744020628.80 total 87.16 last 508.52) - open/close 48808 (time 1744020638.80 total 97.16 last 452.65) - open/close 50000 (time 1744020641.41 total 99.77 last 457.07) - open/close 54969 (time 1744020651.41 total 109.77 last 496.80) - open/close 59642 (time 1744020661.41 total 119.77 last 467.30) - open/close 64479 (time 1744020671.41 total 129.77 last 483.61) total: 64768 open/close in 130.40 seconds: 496.69 ops/second - /unlink 9106 (time 1744020682.85 total 10.00 last 910.52) - /unlink 17844 (time 1744020692.85 total 20.00 last 873.71) - /unlink 20000 (time 1744020695.31 total 22.46 last 877.71) - /unlink 28622 (time 1744020705.31 total 32.46 last 862.19) - /unlink 36779 (time 1744020715.31 total 42.46 last 815.61) - /unlink 40000 (time 1744020718.72 total 45.87 last 944.52) - /unlink 47562 (time 1744020728.72 total 55.87 last 756.19) - /unlink 50000 (time 1744020732.23 total 59.38 last 694.85) - /unlink 58990 (time 1744020742.23 total 69.38 last 898.99) total: 64768 /unlink in 75.61 seconds: 856.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3790 (time 1744020781.99 total 10.00 last 378.99) - open/close 7326 (time 1744020791.99 total 20.00 last 353.57) - open/close 10000 (time 1744020799.72 total 27.73 last 345.81) - open/close 14070 (time 1744020809.72 total 37.73 last 406.95) - open/close 18724 (time 1744020819.72 total 47.74 last 465.35) - open/close 20000 (time 1744020822.42 total 50.43 last 473.54) - open/close 24751 (time 1744020832.42 total 60.43 last 475.03) - open/close 28807 (time 1744020842.42 total 70.43 last 405.50) - open/close 30000 (time 1744020846.14 total 74.15 last 321.00) - open/close 34488 (time 1744020856.14 total 84.15 last 448.69) - open/close 37813 (time 1744020866.14 total 94.15 last 332.48) - open/close 40000 (time 1744020871.42 total 99.43 last 414.50) - open/close 43823 (time 1744020881.42 total 109.43 last 382.27) - open/close 47804 (time 1744020891.44 total 119.45 last 397.27) - open/close 50000 (time 1744020895.89 total 123.90 last 493.68) - open/close 54551 (time 1744020905.89 total 133.90 last 455.06) - open/close 59086 (time 1744020915.89 total 143.90 last 453.45) - open/close 63992 (time 1744020925.89 total 153.90 last 490.57) total: 64768 open/close in 155.94 seconds: 415.34 ops/second - /unlink 7087 (time 1744020939.11 total 10.00 last 708.58) - /unlink 10000 (time 1744020942.32 total 13.22 last 905.34) - /unlink 17703 (time 1744020952.33 total 23.22 last 770.17) - /unlink 20000 (time 1744020956.39 total 27.29 last 565.10) - /unlink 25501 (time 1744020966.39 total 37.29 last 550.07) - /unlink 30000 (time 1744020972.86 total 43.76 last 695.43) - /unlink 36184 (time 1744020982.86 total 53.76 last 618.32) - /unlink 40000 (time 1744020989.17 total 60.06 last 605.12) - /unlink 47118 (time 1744020999.17 total 70.06 last 711.78) - /unlink 50000 (time 1744021002.81 total 73.70 last 792.42) - /unlink 58063 (time 1744021012.81 total 83.70 last 806.18) - /unlink 60000 (time 1744021015.92 total 86.81 last 622.50) total: 64768 /unlink in 93.29 seconds: 694.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2