== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:15:15 (1734257715) Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg457-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1557 (time 1734257773.56 total 10.01 last 155.59) - open/close 3789 (time 1734257783.57 total 20.01 last 223.12) - open/close 6174 (time 1734257793.57 total 30.01 last 238.45) - open/close 8269 (time 1734257803.58 total 40.02 last 209.39) - open/close 10000 (time 1734257812.66 total 49.10 last 190.57) - open/close 11627 (time 1734257822.66 total 59.10 last 162.64) - open/close 13353 (time 1734257832.66 total 69.11 last 172.55) - open/close 15137 (time 1734257842.66 total 79.11 last 178.40) - open/close 17211 (time 1734257852.67 total 89.12 last 207.23) - open/close 19253 (time 1734257862.67 total 99.12 last 204.20) - open/close 20000 (time 1734257866.22 total 102.67 last 210.38) - open/close 21898 (time 1734257876.22 total 112.67 last 189.79) - open/close 23394 (time 1734257886.23 total 122.67 last 149.52) - open/close 24859 (time 1734257896.23 total 132.67 last 146.48) - open/close 26449 (time 1734257906.23 total 142.68 last 158.97) - open/close 28043 (time 1734257916.24 total 152.68 last 159.35) - open/close 29509 (time 1734257926.24 total 162.68 last 146.52) - open/close 30000 (time 1734257929.35 total 165.80 last 157.86) - open/close 31395 (time 1734257939.36 total 175.80 last 139.41) - open/close 32695 (time 1734257949.36 total 185.80 last 129.98) - open/close 34270 (time 1734257959.36 total 195.81 last 157.44) - open/close 35711 (time 1734257969.37 total 205.81 last 144.00) - open/close 37208 (time 1734257979.37 total 215.81 last 149.69) - open/close 38949 (time 1734257989.38 total 225.82 last 174.00) - open/close 40000 (time 1734257995.80 total 232.25 last 163.53) - open/close 41897 (time 1734258005.81 total 242.25 last 189.64) - open/close 43469 (time 1734258015.81 total 252.26 last 157.12) - open/close 44972 (time 1734258025.81 total 262.26 last 150.30) - open/close 46453 (time 1734258035.82 total 272.26 last 148.04) - open/close 47929 (time 1734258045.82 total 282.26 last 147.57) - open/close 49764 (time 1734258055.82 total 292.26 last 183.45) - open/close 51710 (time 1734258065.82 total 302.27 last 194.57) - open/close 54185 (time 1734258075.83 total 312.27 last 247.40) - open/close 56928 (time 1734258085.83 total 322.27 last 274.24) - open/close 59143 (time 1734258095.83 total 332.27 last 221.45) - open/close 60000 (time 1734258100.49 total 336.93 last 183.99) - open/close 62127 (time 1734258110.49 total 346.93 last 212.65) - open/close 64270 (time 1734258120.49 total 356.94 last 214.25) total: 64768 open/close in 359.76 seconds: 180.03 ops/second - /unlink 3815 (time 1734258134.51 total 10.00 last 381.43) - /unlink 8013 (time 1734258144.51 total 20.00 last 419.68) - /unlink 10000 (time 1734258152.44 total 27.93 last 250.56) - /unlink 12264 (time 1734258162.44 total 37.94 last 226.35) - /unlink 14668 (time 1734258172.44 total 47.94 last 240.35) - /unlink 17301 (time 1734258182.45 total 57.94 last 263.25) - /unlink 20000 (time 1734258191.87 total 67.36 last 286.51) - /unlink 22453 (time 1734258201.87 total 77.37 last 245.10) - /unlink 24913 (time 1734258211.87 total 87.37 last 246.00) - /unlink 27501 (time 1734258221.88 total 97.37 last 258.78) - /unlink 30000 (time 1734258230.38 total 105.88 last 293.84) - /unlink 34183 (time 1734258240.38 total 115.88 last 418.24) - /unlink 39101 (time 1734258250.38 total 125.88 last 491.80) - /unlink 44700 (time 1734258260.38 total 135.88 last 559.89) - /unlink 50000 (time 1734258269.76 total 145.25 last 565.25) - /unlink 55618 (time 1734258279.76 total 155.25 last 561.75) - /unlink 60000 (time 1734258289.11 total 164.60 last 468.74) total: 64768 /unlink in 173.73 seconds: 372.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3724 (time 1734258323.62 total 10.00 last 372.33) - open/close 7494 (time 1734258333.62 total 20.00 last 376.97) - open/close 10000 (time 1734258339.20 total 25.58 last 449.63) - open/close 14380 (time 1734258349.20 total 35.58 last 437.95) - open/close 19132 (time 1734258359.20 total 45.58 last 475.05) - open/close 20000 (time 1734258361.51 total 47.89 last 376.60) - open/close 24180 (time 1734258371.51 total 57.89 last 417.91) - open/close 27700 (time 1734258381.51 total 67.89 last 351.96) - open/close 30000 (time 1734258387.88 total 74.26 last 360.85) - open/close 33939 (time 1734258397.89 total 84.26 last 393.83) - open/close 37706 (time 1734258407.89 total 94.27 last 376.64) - open/close 40000 (time 1734258413.88 total 100.26 last 382.77) - open/close 43147 (time 1734258423.88 total 110.26 last 314.65) - open/close 46693 (time 1734258433.91 total 120.29 last 353.53) - open/close 49272 (time 1734258443.92 total 130.29 last 257.84) - open/close 50000 (time 1734258447.30 total 133.68 last 215.19) - open/close 52222 (time 1734258457.30 total 143.68 last 222.20) - open/close 54865 (time 1734258467.30 total 153.68 last 264.19) - open/close 58167 (time 1734258477.30 total 163.68 last 330.13) - open/close 60000 (time 1734258483.67 total 170.04 last 288.11) - open/close 63146 (time 1734258493.67 total 180.05 last 314.49) total: 64768 open/close in 186.25 seconds: 347.75 ops/second - /unlink 5266 (time 1734258510.88 total 10.00 last 526.52) - /unlink 10000 (time 1734258520.10 total 19.22 last 513.39) - /unlink 14943 (time 1734258530.10 total 29.22 last 494.30) - /unlink 20000 (time 1734258538.57 total 37.69 last 596.88) - /unlink 25201 (time 1734258548.57 total 47.70 last 520.07) - /unlink 30000 (time 1734258558.11 total 57.23 last 503.13) - /unlink 35389 (time 1734258568.11 total 67.23 last 538.90) - /unlink 40000 (time 1734258573.99 total 73.12 last 783.41) - /unlink 47331 (time 1734258583.99 total 83.12 last 733.05) - /unlink 50000 (time 1734258588.46 total 87.59 last 597.25) - /unlink 55641 (time 1734258598.46 total 97.59 last 564.02) - /unlink 60000 (time 1734258606.74 total 105.86 last 526.99) total: 64768 /unlink in 115.83 seconds: 559.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3123 (time 1734258643.49 total 10.00 last 312.23) - open/close 6012 (time 1734258653.50 total 20.00 last 288.87) - open/close 9172 (time 1734258663.50 total 30.00 last 315.98) - open/close 10000 (time 1734258665.69 total 32.20 last 377.42) - open/close 13646 (time 1734258675.69 total 42.20 last 364.58) - open/close 17065 (time 1734258685.69 total 52.20 last 341.78) - open/close 20000 (time 1734258695.66 total 62.17 last 294.55) - open/close 22731 (time 1734258705.66 total 72.17 last 273.05) - open/close 25081 (time 1734258715.67 total 82.17 last 234.89) - open/close 27987 (time 1734258725.67 total 92.17 last 290.57) - open/close 30000 (time 1734258732.21 total 98.72 last 307.60) - open/close 33483 (time 1734258742.21 total 108.72 last 348.26) - open/close 36906 (time 1734258752.21 total 118.72 last 342.25) - open/close 40000 (time 1734258760.72 total 127.23 last 363.71) - open/close 43657 (time 1734258770.72 total 137.23 last 365.70) - open/close 46736 (time 1734258780.72 total 147.23 last 307.88) - open/close 50000 (time 1734258790.24 total 156.74 last 343.00) - open/close 52682 (time 1734258800.24 total 166.75 last 268.12) - open/close 55649 (time 1734258810.28 total 176.78 last 295.62) - open/close 58433 (time 1734258820.28 total 186.79 last 278.35) - open/close 60000 (time 1734258825.59 total 192.10 last 294.88) - open/close 63265 (time 1734258835.59 total 202.10 last 326.46) total: 64768 open/close in 209.49 seconds: 309.17 ops/second - /unlink 4379 (time 1734258854.12 total 10.00 last 437.87) - /unlink 9497 (time 1734258864.13 total 20.00 last 511.73) - /unlink 15276 (time 1734258874.13 total 30.00 last 577.87) - /unlink 20000 (time 1734258881.46 total 37.34 last 643.84) - /unlink 25613 (time 1734258891.46 total 47.34 last 561.28) - /unlink 30000 (time 1734258899.31 total 55.18 last 559.52) - /unlink 35855 (time 1734258909.31 total 65.18 last 585.46) - /unlink 40000 (time 1734258917.11 total 72.98 last 531.42) - /unlink 45732 (time 1734258927.11 total 82.98 last 573.16) - /unlink 50000 (time 1734258932.27 total 88.14 last 826.85) - /unlink 57854 (time 1734258942.27 total 98.15 last 785.25) - /unlink 60000 (time 1734258945.60 total 101.48 last 644.19) total: 64768 /unlink in 108.42 seconds: 597.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3209 (time 1734258984.57 total 10.00 last 320.80) - open/close 6323 (time 1734258994.57 total 20.00 last 311.40) - open/close 9508 (time 1734259004.57 total 30.01 last 318.42) - open/close 13227 (time 1734259014.57 total 40.01 last 371.85) - open/close 16454 (time 1734259024.57 total 50.01 last 322.69) - open/close 19628 (time 1734259034.57 total 60.01 last 317.39) - open/close 22593 (time 1734259044.57 total 70.01 last 296.40) - open/close 25444 (time 1734259054.58 total 80.01 last 285.08) - open/close 28286 (time 1734259064.58 total 90.01 last 284.19) - open/close 30000 (time 1734259069.50 total 94.94 last 348.05) - open/close 33545 (time 1734259079.50 total 104.94 last 354.42) - open/close 36759 (time 1734259089.52 total 114.96 last 320.77) - open/close 39458 (time 1734259099.52 total 124.96 last 269.90) - open/close 42837 (time 1734259109.52 total 134.96 last 337.86) - open/close 46381 (time 1734259119.52 total 144.96 last 354.38) - open/close 50000 (time 1734259128.85 total 154.28 last 388.25) - open/close 53859 (time 1734259138.85 total 164.28 last 385.86) - open/close 57463 (time 1734259148.85 total 174.28 last 360.39) - open/close 60000 (time 1734259154.64 total 180.07 last 438.27) - open/close 64035 (time 1734259164.64 total 190.07 last 403.41) total: 64768 open/close in 191.80 seconds: 337.68 ops/second - /unlink 8191 (time 1734259177.25 total 10.00 last 819.09) - /unlink 10000 (time 1734259179.73 total 12.47 last 730.94) - /unlink 18324 (time 1734259189.73 total 22.48 last 832.32) - /unlink 27165 (time 1734259199.73 total 32.48 last 884.09) - /unlink 30000 (time 1734259202.54 total 35.29 last 1007.14) - /unlink 39470 (time 1734259212.54 total 45.29 last 946.95) - /unlink 48976 (time 1734259222.54 total 55.29 last 950.52) - /unlink 59024 (time 1734259232.54 total 65.29 last 1004.70) total: 64768 /unlink in 71.72 seconds: 903.05 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5410 (time 1734259279.93 total 10.00 last 540.89) - open/close 10000 (time 1734259289.24 total 19.32 last 492.74) - open/close 15377 (time 1734259299.24 total 29.32 last 537.69) - open/close 20000 (time 1734259307.20 total 37.28 last 580.67) - open/close 25656 (time 1734259317.20 total 47.28 last 565.54) - open/close 30000 (time 1734259324.73 total 54.80 last 577.34) - open/close 35942 (time 1734259334.73 total 64.81 last 594.11) - open/close 40000 (time 1734259342.03 total 72.11 last 555.91) - open/close 44624 (time 1734259352.03 total 82.11 last 462.36) - open/close 49420 (time 1734259362.03 total 92.11 last 479.56) - open/close 54769 (time 1734259372.03 total 102.11 last 534.79) - open/close 59142 (time 1734259382.03 total 112.11 last 437.22) - open/close 60000 (time 1734259384.08 total 114.16 last 419.36) - open/close 64191 (time 1734259394.08 total 124.16 last 419.03) total: 64768 open/close in 125.56 seconds: 515.82 ops/second - /unlink 7900 (time 1734259406.35 total 10.00 last 789.98) - /unlink 10000 (time 1734259408.98 total 12.63 last 797.10) - /unlink 17579 (time 1734259418.98 total 22.63 last 757.89) - /unlink 20000 (time 1734259421.87 total 25.52 last 838.51) - /unlink 28512 (time 1734259431.87 total 35.52 last 851.12) - /unlink 36941 (time 1734259441.87 total 45.52 last 842.84) - /unlink 40000 (time 1734259445.66 total 49.32 last 806.53) - /unlink 48539 (time 1734259455.66 total 59.32 last 853.84) - /unlink 56935 (time 1734259465.66 total 69.32 last 839.52) - /unlink 60000 (time 1734259469.34 total 73.00 last 833.21) total: 64768 /unlink in 78.62 seconds: 823.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2