== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:17:53 (1734257873) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1431 (time 1734257945.75 total 10.00 last 143.07) - open/close 2947 (time 1734257955.75 total 20.00 last 151.60) - open/close 4370 (time 1734257965.76 total 30.01 last 142.18) - open/close 5945 (time 1734257975.76 total 40.01 last 157.47) - open/close 7524 (time 1734257985.76 total 50.02 last 157.84) - open/close 9059 (time 1734257995.77 total 60.02 last 153.48) - open/close 10000 (time 1734258002.07 total 66.32 last 149.29) - open/close 11179 (time 1734258012.07 total 76.32 last 117.90) - open/close 12325 (time 1734258022.08 total 86.33 last 114.52) - open/close 13734 (time 1734258032.08 total 96.33 last 140.83) - open/close 14886 (time 1734258042.08 total 106.34 last 115.16) - open/close 16057 (time 1734258052.09 total 116.34 last 117.07) - open/close 17404 (time 1734258062.09 total 126.34 last 134.64) - open/close 18743 (time 1734258072.09 total 136.35 last 133.89) - open/close 20000 (time 1734258081.81 total 146.07 last 129.32) - open/close 21303 (time 1734258091.82 total 156.07 last 130.26) - open/close 22516 (time 1734258101.82 total 166.08 last 121.19) - open/close 23722 (time 1734258111.83 total 176.08 last 120.59) - open/close 24931 (time 1734258121.83 total 186.09 last 120.80) - open/close 26180 (time 1734258131.84 total 196.09 last 124.83) - open/close 27351 (time 1734258141.84 total 206.10 last 117.06) - open/close 28737 (time 1734258151.85 total 216.10 last 138.53) - open/close 30000 (time 1734258160.99 total 225.25 last 138.11) - open/close 31160 (time 1734258171.00 total 235.25 last 115.92) - open/close 32511 (time 1734258181.00 total 245.25 last 135.08) - open/close 34099 (time 1734258191.01 total 255.26 last 158.65) - open/close 35322 (time 1734258201.02 total 265.27 last 122.22) - open/close 36697 (time 1734258211.03 total 275.28 last 137.36) - open/close 37942 (time 1734258221.03 total 285.28 last 124.50) - open/close 39365 (time 1734258231.03 total 295.28 last 142.26) - open/close 40000 (time 1734258235.29 total 299.55 last 148.97) - open/close 41395 (time 1734258245.30 total 309.55 last 139.39) - open/close 42667 (time 1734258255.30 total 319.56 last 127.16) - open/close 44118 (time 1734258265.31 total 329.56 last 145.06) - open/close 45415 (time 1734258275.31 total 339.56 last 129.69) - open/close 46746 (time 1734258285.31 total 349.57 last 133.04) - open/close 48263 (time 1734258295.32 total 359.57 last 151.62) - open/close 49594 (time 1734258305.32 total 369.57 last 133.05) - open/close 50000 (time 1734258308.14 total 372.40 last 143.87) - open/close 51364 (time 1734258318.15 total 382.40 last 136.33) - open/close 52625 (time 1734258328.15 total 392.40 last 126.09) - open/close 53904 (time 1734258338.16 total 402.41 last 127.81) - open/close 55262 (time 1734258348.16 total 412.41 last 135.79) - open/close 56593 (time 1734258358.16 total 422.41 last 133.09) - open/close 57970 (time 1734258368.16 total 432.41 last 137.70) - open/close 59314 (time 1734258378.17 total 442.42 last 134.29) - open/close 60000 (time 1734258383.16 total 447.42 last 137.28) - open/close 61446 (time 1734258393.17 total 457.42 last 144.55) - open/close 63134 (time 1734258403.17 total 467.42 last 168.74) - open/close 64727 (time 1734258413.17 total 477.43 last 159.24) total: 64768 open/close in 477.70 seconds: 135.58 ops/second - /unlink 3166 (time 1734258424.77 total 10.00 last 316.50) - /unlink 5683 (time 1734258434.77 total 20.00 last 251.68) - /unlink 8629 (time 1734258444.77 total 30.00 last 294.58) - /unlink 10000 (time 1734258450.48 total 35.71 last 240.23) - /unlink 12494 (time 1734258460.48 total 45.71 last 249.36) - /unlink 15372 (time 1734258470.48 total 55.71 last 287.77) - /unlink 17483 (time 1734258480.48 total 65.72 last 211.07) - /unlink 19722 (time 1734258490.48 total 75.72 last 223.87) - /unlink 22351 (time 1734258500.48 total 85.72 last 262.89) - /unlink 24497 (time 1734258510.48 total 95.72 last 214.57) - /unlink 26846 (time 1734258520.49 total 105.72 last 234.81) - /unlink 29144 (time 1734258530.49 total 115.72 last 229.80) - /unlink 30000 (time 1734258535.27 total 120.51 last 178.83) - /unlink 32059 (time 1734258545.28 total 130.51 last 205.80) - /unlink 34516 (time 1734258555.28 total 140.51 last 245.70) - /unlink 36560 (time 1734258565.28 total 150.51 last 204.39) - /unlink 38697 (time 1734258575.29 total 160.52 last 213.58) - /unlink 40000 (time 1734258580.40 total 165.64 last 254.62) - /unlink 42286 (time 1734258590.40 total 175.64 last 228.56) - /unlink 44772 (time 1734258600.41 total 185.64 last 248.56) - /unlink 47254 (time 1734258610.41 total 195.65 last 248.11) - /unlink 49629 (time 1734258620.41 total 205.65 last 237.48) - /unlink 52072 (time 1734258630.41 total 215.65 last 244.24) - /unlink 54158 (time 1734258640.42 total 225.65 last 208.52) - /unlink 56484 (time 1734258650.42 total 235.65 last 232.59) - /unlink 58899 (time 1734258660.42 total 245.66 last 241.37) - /unlink 60000 (time 1734258665.54 total 250.77 last 215.35) - /unlink 62137 (time 1734258675.54 total 260.77 last 213.63) total: 64768 /unlink in 269.45 seconds: 240.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1792 (time 1734258712.32 total 10.00 last 179.14) - open/close 3198 (time 1734258722.32 total 20.00 last 140.58) - open/close 4616 (time 1734258732.33 total 30.01 last 141.76) - open/close 6072 (time 1734258742.33 total 40.01 last 145.58) - open/close 7420 (time 1734258752.33 total 50.01 last 134.77) - open/close 9292 (time 1734258762.33 total 60.01 last 187.19) - open/close 10000 (time 1734258766.44 total 64.12 last 172.14) - open/close 11679 (time 1734258776.45 total 74.13 last 167.84) - open/close 13319 (time 1734258786.45 total 84.13 last 163.92) - open/close 15205 (time 1734258796.46 total 94.14 last 188.48) - open/close 16835 (time 1734258806.46 total 104.14 last 162.93) - open/close 18152 (time 1734258816.46 total 114.14 last 131.70) - open/close 19737 (time 1734258826.47 total 124.15 last 158.46) - open/close 21308 (time 1734258836.47 total 134.15 last 157.06) - open/close 23009 (time 1734258846.47 total 144.15 last 170.09) - open/close 24820 (time 1734258856.47 total 154.15 last 181.09) - open/close 26895 (time 1734258866.47 total 164.15 last 207.42) - open/close 29294 (time 1734258876.48 total 174.16 last 239.83) - open/close 30000 (time 1734258880.39 total 178.07 last 180.28) - open/close 31716 (time 1734258890.39 total 188.07 last 171.60) - open/close 33551 (time 1734258900.40 total 198.08 last 183.30) - open/close 35142 (time 1734258910.41 total 208.09 last 159.04) - open/close 36699 (time 1734258920.41 total 218.09 last 155.64) - open/close 38361 (time 1734258930.43 total 228.11 last 165.89) - open/close 40000 (time 1734258939.02 total 236.70 last 190.88) - open/close 41819 (time 1734258949.02 total 246.70 last 181.90) - open/close 43965 (time 1734258959.02 total 256.70 last 214.55) - open/close 45908 (time 1734258969.02 total 266.70 last 194.28) - open/close 47753 (time 1734258979.02 total 276.70 last 184.48) - open/close 49671 (time 1734258989.02 total 286.70 last 191.74) - open/close 51777 (time 1734258999.03 total 296.71 last 210.51) - open/close 53949 (time 1734259009.03 total 306.71 last 217.11) - open/close 55982 (time 1734259019.04 total 316.72 last 203.21) - open/close 57700 (time 1734259029.04 total 326.72 last 171.78) - open/close 59365 (time 1734259039.04 total 336.72 last 166.49) - open/close 60000 (time 1734259042.54 total 340.23 last 181.09) - open/close 61971 (time 1734259052.55 total 350.23 last 197.07) - open/close 63695 (time 1734259062.55 total 360.23 last 172.34) total: 64768 open/close in 365.79 seconds: 177.06 ops/second - /unlink 3570 (time 1734259079.39 total 10.00 last 356.96) - /unlink 6542 (time 1734259089.39 total 20.00 last 297.09) - /unlink 9653 (time 1734259099.39 total 30.01 last 310.99) - /unlink 12600 (time 1734259109.39 total 40.01 last 294.70) - /unlink 16339 (time 1734259119.39 total 50.01 last 373.89) - /unlink 19722 (time 1734259129.40 total 60.01 last 338.24) - /unlink 22808 (time 1734259139.40 total 70.01 last 308.48) - /unlink 26214 (time 1734259149.40 total 80.02 last 340.53) - /unlink 29311 (time 1734259159.40 total 90.02 last 309.69) - /unlink 30000 (time 1734259161.47 total 92.09 last 332.65) - /unlink 33495 (time 1734259171.48 total 102.09 last 349.43) - /unlink 36554 (time 1734259181.48 total 112.09 last 305.87) - /unlink 40000 (time 1734259191.33 total 121.94 last 349.89) - /unlink 43168 (time 1734259201.33 total 131.94 last 316.77) - /unlink 46412 (time 1734259211.33 total 141.94 last 324.40) - /unlink 50000 (time 1734259221.00 total 151.62 last 370.76) - /unlink 52675 (time 1734259231.01 total 161.62 last 267.41) - /unlink 55681 (time 1734259241.01 total 171.62 last 300.56) - /unlink 59172 (time 1734259251.01 total 181.62 last 349.04) - /unlink 60000 (time 1734259253.25 total 183.86 last 370.36) - /unlink 63438 (time 1734259263.25 total 193.86 last 343.79) total: 64768 /unlink in 197.34 seconds: 328.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2350 (time 1734259293.84 total 10.00 last 234.95) - open/close 4154 (time 1734259303.84 total 20.00 last 180.36) - open/close 6024 (time 1734259313.84 total 30.01 last 186.94) - open/close 7640 (time 1734259323.84 total 40.01 last 161.57) - open/close 9460 (time 1734259333.85 total 50.02 last 181.90) - open/close 10000 (time 1734259336.44 total 52.60 last 208.53) - open/close 12041 (time 1734259346.44 total 62.61 last 204.06) - open/close 14179 (time 1734259356.44 total 72.61 last 213.75) - open/close 16361 (time 1734259366.45 total 82.61 last 218.15) - open/close 18176 (time 1734259376.45 total 92.61 last 181.46) - open/close 20000 (time 1734259384.72 total 100.88 last 220.58) - open/close 21914 (time 1734259394.72 total 110.89 last 191.35) - open/close 23865 (time 1734259404.72 total 120.89 last 195.09) - open/close 25891 (time 1734259414.72 total 130.89 last 202.56) - open/close 27729 (time 1734259424.73 total 140.89 last 183.72) - open/close 29620 (time 1734259434.73 total 150.89 last 189.10) - open/close 30000 (time 1734259436.91 total 153.08 last 173.75) - open/close 31676 (time 1734259446.91 total 163.08 last 167.58) - open/close 33351 (time 1734259456.92 total 173.09 last 167.41) - open/close 35236 (time 1734259466.92 total 183.09 last 188.43) - open/close 37324 (time 1734259476.92 total 193.09 last 208.78) - open/close 39211 (time 1734259486.93 total 203.09 last 188.62) - open/close 40000 (time 1734259491.16 total 207.33 last 186.27) - open/close 41941 (time 1734259501.17 total 217.33 last 194.02) - open/close 43741 (time 1734259511.17 total 227.33 last 179.99) - open/close 45725 (time 1734259521.17 total 237.34 last 198.36) - open/close 47582 (time 1734259531.17 total 247.34 last 185.69) - open/close 49359 (time 1734259541.17 total 257.34 last 177.69) - open/close 50000 (time 1734259544.84 total 261.00 last 174.80) - open/close 51852 (time 1734259554.84 total 271.01 last 185.16) - open/close 53753 (time 1734259564.84 total 281.01 last 190.06) - open/close 55728 (time 1734259574.85 total 291.01 last 197.46) - open/close 57550 (time 1734259584.85 total 301.01 last 182.13) - open/close 59667 (time 1734259594.85 total 311.02 last 211.63) - open/close 61744 (time 1734259604.86 total 321.02 last 207.61) - open/close 63837 (time 1734259614.86 total 331.03 last 209.20) total: 64768 open/close in 335.50 seconds: 193.05 ops/second - /unlink 4095 (time 1734259630.52 total 10.00 last 409.43) - /unlink 8754 (time 1734259640.52 total 20.00 last 465.84) - /unlink 10000 (time 1734259643.41 total 22.89 last 431.10) - /unlink 14143 (time 1734259653.41 total 32.89 last 414.23) - /unlink 18061 (time 1734259663.42 total 42.90 last 391.70) - /unlink 20000 (time 1734259669.36 total 48.84 last 326.33) - /unlink 23525 (time 1734259679.36 total 58.84 last 352.48) - /unlink 27580 (time 1734259689.36 total 68.84 last 405.41) - /unlink 30000 (time 1734259694.98 total 74.46 last 430.42) - /unlink 34867 (time 1734259704.98 total 84.46 last 486.70) - /unlink 39163 (time 1734259714.98 total 94.46 last 429.59) - /unlink 40000 (time 1734259717.53 total 97.01 last 328.73) - /unlink 43839 (time 1734259727.53 total 107.01 last 383.89) - /unlink 47523 (time 1734259737.53 total 117.01 last 368.35) - /unlink 50000 (time 1734259745.57 total 125.05 last 308.07) - /unlink 53812 (time 1734259755.58 total 135.06 last 381.07) - /unlink 57034 (time 1734259765.58 total 145.06 last 322.09) - /unlink 60000 (time 1734259774.86 total 154.34 last 319.58) - /unlink 63371 (time 1734259784.86 total 164.35 last 336.96) total: 64768 /unlink in 168.18 seconds: 385.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2360 (time 1734259818.72 total 10.00 last 235.94) - open/close 4821 (time 1734259828.72 total 20.01 last 246.04) - open/close 7085 (time 1734259838.73 total 30.01 last 226.35) - open/close 9476 (time 1734259848.73 total 40.01 last 239.03) - open/close 10000 (time 1734259851.49 total 42.77 last 189.92) - open/close 11776 (time 1734259861.49 total 52.77 last 177.53) - open/close 14014 (time 1734259871.49 total 62.77 last 223.78) - open/close 15854 (time 1734259881.49 total 72.77 last 183.99) - open/close 18081 (time 1734259891.49 total 82.78 last 222.65) - open/close 20000 (time 1734259900.13 total 91.41 last 222.33) - open/close 22181 (time 1734259910.13 total 101.41 last 218.07) - open/close 24393 (time 1734259920.13 total 111.41 last 221.13) - open/close 27176 (time 1734259930.13 total 121.41 last 278.28) - open/close 30000 (time 1734259939.94 total 131.22 last 287.96) - open/close 32784 (time 1734259949.94 total 141.22 last 278.38) - open/close 35591 (time 1734259959.94 total 151.22 last 280.63) - open/close 38430 (time 1734259969.94 total 161.22 last 283.90) - open/close 40000 (time 1734259975.37 total 166.65 last 289.10) - open/close 43009 (time 1734259985.37 total 176.66 last 300.85) - open/close 45714 (time 1734259995.37 total 186.66 last 270.49) - open/close 48561 (time 1734260005.37 total 196.66 last 284.69) - open/close 50000 (time 1734260010.33 total 201.61 last 290.43) - open/close 52727 (time 1734260020.34 total 211.62 last 272.39) - open/close 55748 (time 1734260030.34 total 221.62 last 302.10) - open/close 58643 (time 1734260040.34 total 231.62 last 289.50) - open/close 60000 (time 1734260045.41 total 236.70 last 267.52) - open/close 62534 (time 1734260055.42 total 246.70 last 253.34) total: 64768 open/close in 254.02 seconds: 254.97 ops/second - /unlink 6229 (time 1734260073.70 total 10.00 last 622.84) - /unlink 10000 (time 1734260080.65 total 16.95 last 542.36) - /unlink 15148 (time 1734260090.65 total 26.96 last 514.70) - /unlink 20000 (time 1734260099.86 total 36.16 last 527.29) - /unlink 25134 (time 1734260109.86 total 46.16 last 513.37) - /unlink 30000 (time 1734260119.06 total 55.36 last 528.87) - /unlink 35191 (time 1734260129.06 total 65.36 last 519.02) - /unlink 39979 (time 1734260139.06 total 75.36 last 478.79) - /unlink 45326 (time 1734260149.06 total 85.36 last 534.59) - /unlink 50000 (time 1734260157.88 total 94.18 last 529.85) - /unlink 55682 (time 1734260167.88 total 104.19 last 568.11) - /unlink 60000 (time 1734260175.78 total 112.08 last 546.68) total: 64768 /unlink in 121.56 seconds: 532.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2835 (time 1734260215.00 total 10.00 last 283.49) - open/close 5661 (time 1734260225.01 total 20.00 last 282.51) - open/close 8026 (time 1734260235.01 total 30.01 last 236.47) - open/close 10000 (time 1734260241.68 total 36.68 last 295.61) - open/close 12898 (time 1734260251.69 total 46.69 last 289.73) - open/close 15674 (time 1734260261.69 total 56.69 last 277.54) - open/close 18711 (time 1734260271.69 total 66.69 last 303.67) - open/close 20000 (time 1734260275.72 total 70.72 last 320.00) - open/close 22995 (time 1734260285.72 total 80.72 last 299.41) - open/close 26105 (time 1734260295.72 total 90.72 last 310.91) - open/close 28678 (time 1734260305.73 total 100.73 last 257.13) - open/close 30000 (time 1734260311.39 total 106.39 last 233.65) - open/close 32509 (time 1734260321.39 total 116.39 last 250.89) - open/close 35175 (time 1734260331.39 total 126.39 last 266.52) - open/close 37598 (time 1734260341.39 total 136.39 last 242.26) - open/close 39965 (time 1734260351.40 total 146.40 last 236.63) - open/close 42395 (time 1734260361.40 total 156.40 last 242.99) - open/close 45038 (time 1734260371.40 total 166.40 last 264.28) - open/close 47587 (time 1734260381.40 total 176.40 last 254.87) - open/close 50000 (time 1734260391.00 total 186.00 last 251.29) - open/close 52949 (time 1734260401.00 total 196.00 last 294.88) - open/close 55757 (time 1734260411.01 total 206.00 last 280.72) - open/close 58407 (time 1734260421.01 total 216.01 last 264.95) - open/close 60000 (time 1734260426.92 total 221.91 last 269.57) - open/close 62622 (time 1734260436.92 total 231.92 last 262.17) total: 64768 open/close in 240.18 seconds: 269.66 ops/second - /unlink 6309 (time 1734260456.14 total 10.00 last 630.78) - /unlink 10000 (time 1734260462.41 total 16.27 last 589.09) - /unlink 15470 (time 1734260472.41 total 26.27 last 546.92) - /unlink 20000 (time 1734260481.52 total 35.38 last 497.39) - /unlink 24430 (time 1734260491.52 total 45.38 last 442.95) - /unlink 29624 (time 1734260501.52 total 55.38 last 519.38) - /unlink 35369 (time 1734260511.52 total 65.38 last 574.46) - /unlink 40000 (time 1734260521.29 total 75.15 last 474.11) - /unlink 44775 (time 1734260531.29 total 85.15 last 477.46) - /unlink 49732 (time 1734260541.29 total 95.15 last 495.67) - /unlink 54624 (time 1734260551.29 total 105.15 last 489.16) - /unlink 59603 (time 1734260561.29 total 115.15 last 497.86) total: 64768 /unlink in 124.93 seconds: 518.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2