== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:12 (1734257712) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 oleg420-server: oleg420-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg420-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1790 (time 1734257770.29 total 10.01 last 178.84) - open/close 4136 (time 1734257780.29 total 20.01 last 234.59) - open/close 6754 (time 1734257790.29 total 30.01 last 261.78) - open/close 9399 (time 1734257800.29 total 40.01 last 264.39) - open/close 10000 (time 1734257803.46 total 43.18 last 189.59) - open/close 12001 (time 1734257813.47 total 53.19 last 200.02) - open/close 13833 (time 1734257823.47 total 63.19 last 183.12) - open/close 15716 (time 1734257833.48 total 73.20 last 188.18) - open/close 17725 (time 1734257843.48 total 83.20 last 200.86) - open/close 19975 (time 1734257853.48 total 93.20 last 224.91) - open/close 22237 (time 1734257863.48 total 103.21 last 226.14) - open/close 24427 (time 1734257873.49 total 113.21 last 218.99) - open/close 26183 (time 1734257883.49 total 123.22 last 175.43) - open/close 27704 (time 1734257893.50 total 133.22 last 152.07) - open/close 29548 (time 1734257903.50 total 143.22 last 184.31) - open/close 30000 (time 1734257906.20 total 145.92 last 167.76) - open/close 31792 (time 1734257916.20 total 155.92 last 179.20) - open/close 33436 (time 1734257926.20 total 165.92 last 164.35) - open/close 35049 (time 1734257936.20 total 175.92 last 161.28) - open/close 36489 (time 1734257946.20 total 185.93 last 143.96) - open/close 38249 (time 1734257956.20 total 195.93 last 175.98) - open/close 39950 (time 1734257966.21 total 205.93 last 170.08) - open/close 41639 (time 1734257976.21 total 215.93 last 168.85) - open/close 43523 (time 1734257986.21 total 225.93 last 188.35) - open/close 45316 (time 1734257996.22 total 235.94 last 179.16) - open/close 47387 (time 1734258006.22 total 245.95 last 207.02) - open/close 49048 (time 1734258016.23 total 255.95 last 166.04) - open/close 50000 (time 1734258022.06 total 261.79 last 163.13) - open/close 51732 (time 1734258032.07 total 271.79 last 173.15) - open/close 53288 (time 1734258042.07 total 281.79 last 155.59) - open/close 55395 (time 1734258052.07 total 291.79 last 210.69) - open/close 57344 (time 1734258062.07 total 301.79 last 194.87) - open/close 59959 (time 1734258072.07 total 311.79 last 261.48) - open/close 62752 (time 1734258082.07 total 321.79 last 279.22) total: 64768 open/close in 328.25 seconds: 197.31 ops/second - /unlink 3596 (time 1734258099.56 total 10.00 last 359.59) - /unlink 7735 (time 1734258109.57 total 20.00 last 413.88) - /unlink 10000 (time 1734258114.98 total 25.41 last 418.44) - /unlink 14329 (time 1734258124.98 total 35.42 last 432.71) - /unlink 18651 (time 1734258134.98 total 45.42 last 432.19) - /unlink 20000 (time 1734258138.30 total 48.74 last 406.30) - /unlink 24192 (time 1734258148.30 total 58.74 last 419.16) - /unlink 26807 (time 1734258158.31 total 68.74 last 261.43) - /unlink 29584 (time 1734258168.31 total 78.75 last 277.61) - /unlink 32362 (time 1734258178.31 total 88.75 last 277.67) - /unlink 35384 (time 1734258188.31 total 98.75 last 302.18) - /unlink 38364 (time 1734258198.32 total 108.75 last 297.87) - /unlink 40000 (time 1734258204.70 total 115.14 last 256.34) - /unlink 42672 (time 1734258214.70 total 125.14 last 267.12) - /unlink 45423 (time 1734258224.70 total 135.14 last 275.09) - /unlink 49218 (time 1734258234.71 total 145.14 last 379.37) - /unlink 50000 (time 1734258236.73 total 147.17 last 386.74) - /unlink 55204 (time 1734258246.73 total 157.17 last 520.37) - /unlink 60000 (time 1734258255.37 total 165.80 last 555.39) total: 64768 /unlink in 173.31 seconds: 373.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3100 (time 1734258286.68 total 10.01 last 309.79) - open/close 5701 (time 1734258296.68 total 20.01 last 260.07) - open/close 8995 (time 1734258306.68 total 30.01 last 329.36) - open/close 10000 (time 1734258309.22 total 32.54 last 396.62) - open/close 13968 (time 1734258319.22 total 42.54 last 396.80) - open/close 17466 (time 1734258329.22 total 52.55 last 349.73) - open/close 20000 (time 1734258335.05 total 58.37 last 434.68) - open/close 24717 (time 1734258345.05 total 68.38 last 471.59) - open/close 29227 (time 1734258355.05 total 78.38 last 450.98) - open/close 33624 (time 1734258365.05 total 88.38 last 439.63) - open/close 37291 (time 1734258375.05 total 98.38 last 366.66) - open/close 40000 (time 1734258382.75 total 106.08 last 351.84) - open/close 43613 (time 1734258392.75 total 116.08 last 361.24) - open/close 47075 (time 1734258402.76 total 126.08 last 346.11) - open/close 50000 (time 1734258409.45 total 132.78 last 436.96) - open/close 53349 (time 1734258419.46 total 142.78 last 334.77) - open/close 56624 (time 1734258429.46 total 152.78 last 327.45) - open/close 59841 (time 1734258439.46 total 162.79 last 321.63) - open/close 62215 (time 1734258449.46 total 172.79 last 237.40) - open/close 64510 (time 1734258459.46 total 182.79 last 229.42) total: 64768 open/close in 183.86 seconds: 352.26 ops/second - /unlink 5581 (time 1734258471.67 total 10.00 last 558.07) - /unlink 10000 (time 1734258478.79 total 17.13 last 619.85) - /unlink 15622 (time 1734258488.80 total 27.13 last 562.18) - /unlink 20000 (time 1734258497.17 total 35.51 last 522.63) - /unlink 25475 (time 1734258507.17 total 45.51 last 547.39) - /unlink 30000 (time 1734258516.41 total 54.74 last 489.93) - /unlink 35041 (time 1734258526.41 total 64.75 last 504.03) - /unlink 40000 (time 1734258535.13 total 73.46 last 568.98) - /unlink 45205 (time 1734258545.13 total 83.46 last 520.39) - /unlink 50000 (time 1734258554.26 total 92.60 last 524.97) - /unlink 54735 (time 1734258564.26 total 102.60 last 473.44) - /unlink 60000 (time 1734258571.75 total 110.09 last 703.33) total: 64768 /unlink in 116.06 seconds: 558.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2865 (time 1734258603.21 total 10.00 last 286.49) - open/close 5509 (time 1734258613.21 total 20.00 last 264.33) - open/close 8056 (time 1734258623.21 total 30.00 last 254.68) - open/close 10000 (time 1734258629.26 total 36.05 last 321.49) - open/close 13379 (time 1734258639.26 total 46.05 last 337.78) - open/close 16093 (time 1734258649.27 total 56.06 last 271.35) - open/close 19198 (time 1734258659.27 total 66.06 last 310.49) - open/close 20000 (time 1734258661.81 total 68.60 last 315.51) - open/close 23584 (time 1734258671.81 total 78.60 last 358.38) - open/close 27040 (time 1734258681.81 total 88.60 last 345.56) - open/close 29854 (time 1734258691.81 total 98.60 last 281.38) - open/close 32907 (time 1734258701.81 total 108.60 last 305.27) - open/close 35252 (time 1734258711.82 total 118.61 last 234.40) - open/close 37842 (time 1734258721.82 total 128.61 last 258.93) - open/close 40000 (time 1734258729.31 total 136.10 last 287.96) - open/close 43463 (time 1734258739.31 total 146.10 last 346.28) - open/close 46897 (time 1734258749.32 total 156.11 last 343.31) - open/close 50000 (time 1734258758.51 total 165.30 last 337.40) - open/close 53606 (time 1734258768.51 total 175.30 last 360.56) - open/close 56641 (time 1734258778.51 total 185.30 last 303.46) - open/close 60000 (time 1734258787.95 total 194.74 last 355.84) - open/close 62758 (time 1734258797.95 total 204.74 last 275.79) total: 64768 open/close in 211.88 seconds: 305.68 ops/second - /unlink 5078 (time 1734258816.11 total 10.00 last 507.79) - /unlink 10000 (time 1734258825.12 total 19.01 last 546.26) - /unlink 16092 (time 1734258835.13 total 29.02 last 608.73) - /unlink 19943 (time 1734258845.13 total 39.02 last 385.07) - /unlink 24429 (time 1734258855.13 total 49.02 last 448.48) - /unlink 29662 (time 1734258865.13 total 59.02 last 523.24) - /unlink 35331 (time 1734258875.13 total 69.02 last 566.81) - /unlink 40000 (time 1734258882.49 total 76.39 last 634.11) - /unlink 45535 (time 1734258892.49 total 86.39 last 553.47) - /unlink 50000 (time 1734258901.04 total 94.94 last 522.35) - /unlink 55886 (time 1734258911.04 total 104.94 last 588.58) - /unlink 60000 (time 1734258919.00 total 112.89 last 517.16) total: 64768 /unlink in 121.06 seconds: 534.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3584 (time 1734258953.30 total 10.00 last 358.30) - open/close 7417 (time 1734258963.30 total 20.00 last 383.29) - open/close 10000 (time 1734258970.94 total 27.64 last 338.12) - open/close 13413 (time 1734258980.94 total 37.64 last 341.22) - open/close 16638 (time 1734258990.94 total 47.65 last 322.47) - open/close 19930 (time 1734259000.94 total 57.65 last 329.20) - open/close 23543 (time 1734259010.94 total 67.65 last 361.28) - open/close 26931 (time 1734259020.94 total 77.65 last 338.70) - open/close 30000 (time 1734259030.36 total 87.07 last 325.81) - open/close 33071 (time 1734259040.37 total 97.07 last 307.04) - open/close 36046 (time 1734259050.37 total 107.07 last 297.45) - open/close 38989 (time 1734259060.37 total 117.08 last 294.23) - open/close 40000 (time 1734259063.69 total 120.39 last 305.00) - open/close 43622 (time 1734259073.69 total 130.39 last 362.15) - open/close 47363 (time 1734259083.69 total 140.39 last 374.09) - open/close 50000 (time 1734259091.54 total 148.25 last 335.73) - open/close 52843 (time 1734259101.54 total 158.25 last 284.26) - open/close 56569 (time 1734259111.54 total 168.25 last 372.59) - open/close 60000 (time 1734259120.83 total 177.54 last 369.34) - open/close 64041 (time 1734259130.83 total 187.54 last 404.03) total: 64768 open/close in 189.35 seconds: 342.06 ops/second - /unlink 7375 (time 1734259143.53 total 10.00 last 737.47) - /unlink 10000 (time 1734259147.50 total 13.97 last 661.14) - /unlink 18138 (time 1734259157.50 total 23.97 last 813.79) - /unlink 20000 (time 1734259159.86 total 26.32 last 791.23) - /unlink 28095 (time 1734259169.86 total 36.32 last 809.48) - /unlink 30000 (time 1734259172.04 total 38.51 last 873.41) - /unlink 37394 (time 1734259182.04 total 48.51 last 739.39) - /unlink 40000 (time 1734259185.46 total 51.93 last 760.97) - /unlink 48487 (time 1734259195.46 total 61.93 last 848.62) - /unlink 58042 (time 1734259205.46 total 71.93 last 955.48) - /unlink 60000 (time 1734259207.47 total 73.94 last 973.92) total: 64768 /unlink in 79.29 seconds: 816.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5203 (time 1734259244.00 total 10.00 last 520.22) - open/close 10000 (time 1734259253.77 total 19.77 last 491.19) - open/close 15442 (time 1734259263.77 total 29.77 last 544.13) - open/close 20000 (time 1734259271.42 total 37.42 last 595.69) - open/close 25456 (time 1734259281.42 total 47.42 last 545.58) - open/close 30000 (time 1734259290.12 total 56.12 last 522.17) - open/close 35535 (time 1734259300.13 total 66.12 last 553.45) - open/close 40000 (time 1734259307.50 total 73.50 last 605.16) - open/close 45831 (time 1734259317.51 total 83.51 last 582.81) - open/close 50000 (time 1734259324.71 total 90.71 last 578.95) - open/close 55873 (time 1734259334.71 total 100.71 last 587.28) - open/close 60000 (time 1734259342.29 total 108.29 last 544.32) - open/close 64559 (time 1734259352.29 total 118.29 last 455.86) total: 64768 open/close in 118.68 seconds: 545.75 ops/second - /unlink 8606 (time 1734259363.54 total 10.00 last 860.54) - /unlink 17681 (time 1734259373.54 total 20.00 last 907.44) - /unlink 20000 (time 1734259376.38 total 22.84 last 818.13) - /unlink 26824 (time 1734259386.38 total 32.84 last 682.37) - /unlink 30000 (time 1734259390.89 total 37.35 last 703.85) - /unlink 37181 (time 1734259400.89 total 47.35 last 718.06) - /unlink 40000 (time 1734259404.80 total 51.26 last 720.84) - /unlink 47317 (time 1734259414.80 total 61.26 last 731.68) - /unlink 50000 (time 1734259418.94 total 65.41 last 647.29) - /unlink 58208 (time 1734259428.95 total 75.41 last 820.77) - /unlink 60000 (time 1734259431.24 total 77.70 last 781.11) total: 64768 /unlink in 83.20 seconds: 778.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2