== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:03:40 (1777421020) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32583 327 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1344 (time 1777421085.70 total 10.00 last 134.37) - open/close 2666 (time 1777421095.70 total 20.00 last 132.19) - open/close 4149 (time 1777421105.71 total 30.01 last 148.27) - open/close 5490 (time 1777421115.71 total 40.01 last 134.05) - open/close 6733 (time 1777421125.72 total 50.01 last 124.24) - open/close 8062 (time 1777421135.72 total 60.02 last 132.87) - open/close 9091 (time 1777421145.72 total 70.02 last 102.87) - open/close 10000 (time 1777421155.65 total 79.95 last 91.52) - open/close 10955 (time 1777421165.66 total 89.95 last 95.46) - open/close 11855 (time 1777421175.67 total 99.97 last 89.90) - open/close 12759 (time 1777421185.67 total 109.97 last 90.35) - open/close 13681 (time 1777421195.68 total 119.98 last 92.15) - open/close 14586 (time 1777421205.68 total 129.98 last 90.48) - open/close 15531 (time 1777421215.69 total 139.99 last 94.43) - open/close 16583 (time 1777421225.69 total 149.99 last 105.18) - open/close 17735 (time 1777421235.69 total 159.99 last 115.19) - open/close 18920 (time 1777421245.69 total 169.99 last 118.48) - open/close 20000 (time 1777421255.47 total 179.77 last 110.47) - open/close 21211 (time 1777421265.47 total 189.77 last 121.08) - open/close 22396 (time 1777421275.48 total 199.77 last 118.45) - open/close 23428 (time 1777421285.48 total 209.78 last 103.18) - open/close 24591 (time 1777421295.48 total 219.78 last 116.27) - open/close 25718 (time 1777421305.49 total 229.79 last 112.54) - open/close 26726 (time 1777421315.51 total 239.80 last 100.68) - open/close 27698 (time 1777421325.51 total 249.81 last 97.12) - open/close 28759 (time 1777421335.52 total 259.81 last 106.08) - open/close 29881 (time 1777421345.53 total 269.83 last 112.07) - open/close 31028 (time 1777421355.53 total 279.83 last 114.67) - open/close 32058 (time 1777421365.53 total 289.83 last 102.95) - open/close 33094 (time 1777421375.53 total 299.83 last 103.60) - open/close 34113 (time 1777421385.55 total 309.85 last 101.75) - open/close 35101 (time 1777421395.55 total 319.85 last 98.78) - open/close 36000 (time 1777421405.56 total 329.86 last 89.85) - open/close 36977 (time 1777421415.57 total 339.86 last 97.61) - open/close 38001 (time 1777421425.57 total 349.87 last 102.39) - open/close 39175 (time 1777421435.57 total 359.87 last 117.37) - open/close 40000 (time 1777421442.75 total 367.04 last 114.97) - open/close 41200 (time 1777421452.75 total 377.05 last 119.92) - open/close 42411 (time 1777421462.76 total 387.05 last 121.05) - open/close 43803 (time 1777421472.76 total 397.06 last 139.17) - open/close 45162 (time 1777421482.76 total 407.06 last 135.89) - open/close 46724 (time 1777421492.76 total 417.06 last 156.17) - open/close 48131 (time 1777421502.76 total 427.06 last 140.69) - open/close 49432 (time 1777421512.76 total 437.06 last 130.06) - open/close 50000 (time 1777421516.96 total 441.26 last 135.48) - open/close 51462 (time 1777421526.96 total 451.26 last 146.12) - open/close 52707 (time 1777421536.96 total 461.26 last 124.48) - open/close 54075 (time 1777421546.97 total 471.27 last 136.76) - open/close 55495 (time 1777421556.98 total 481.27 last 141.89) - open/close 56591 (time 1777421566.98 total 491.28 last 109.57) - open/close 57897 (time 1777421576.98 total 501.28 last 130.54) - open/close 59292 (time 1777421586.99 total 511.29 last 139.33) - open/close 60000 (time 1777421592.41 total 516.71 last 130.69) - open/close 61554 (time 1777421602.41 total 526.71 last 155.38) - open/close 63855 (time 1777421612.41 total 536.71 last 230.08) total: 64768 open/close in 541.29 seconds: 119.65 ops/second - /unlink 3822 (time 1777421628.66 total 10.00 last 382.09) - /unlink 7695 (time 1777421638.66 total 20.00 last 387.25) - /unlink 10000 (time 1777421644.27 total 25.62 last 410.68) - /unlink 13778 (time 1777421654.27 total 35.62 last 377.71) - /unlink 17591 (time 1777421664.27 total 45.62 last 381.23) - /unlink 20000 (time 1777421671.53 total 52.87 last 332.14) - /unlink 23413 (time 1777421681.53 total 62.88 last 341.20) - /unlink 26667 (time 1777421691.53 total 72.88 last 325.40) - /unlink 30000 (time 1777421700.79 total 82.13 last 360.09) - /unlink 33678 (time 1777421710.79 total 92.14 last 367.64) - /unlink 37229 (time 1777421720.79 total 102.14 last 355.05) - /unlink 40000 (time 1777421726.24 total 107.59 last 508.44) - /unlink 43837 (time 1777421736.25 total 117.59 last 383.59) - /unlink 47569 (time 1777421746.25 total 127.59 last 373.19) - /unlink 50000 (time 1777421751.95 total 133.29 last 426.32) - /unlink 53529 (time 1777421761.95 total 143.30 last 352.85) - /unlink 57191 (time 1777421771.95 total 153.30 last 366.20) - /unlink 60000 (time 1777421778.43 total 159.78 last 433.18) total: 64768 /unlink in 169.57 seconds: 381.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 882 (time 1777421825.15 total 10.01 last 88.13) - open/close 1844 (time 1777421835.17 total 20.03 last 95.97) - open/close 2561 (time 1777421845.18 total 30.04 last 71.66) - open/close 3153 (time 1777421855.18 total 40.04 last 59.18) - open/close 4005 (time 1777421865.20 total 50.05 last 85.09) - open/close 4976 (time 1777421875.20 total 60.06 last 97.05) - open/close 5989 (time 1777421885.21 total 70.07 last 101.21) - open/close 7065 (time 1777421895.22 total 80.08 last 107.49) - open/close 8203 (time 1777421905.22 total 90.08 last 113.75) - open/close 9316 (time 1777421915.23 total 100.09 last 111.19) - open/close 10000 (time 1777421921.77 total 106.63 last 104.59) - open/close 10987 (time 1777421931.78 total 116.63 last 98.68) - open/close 11943 (time 1777421941.79 total 126.64 last 95.51) - open/close 13073 (time 1777421951.79 total 136.65 last 112.90) - open/close 14175 (time 1777421961.80 total 146.66 last 110.10) - open/close 15262 (time 1777421971.81 total 156.67 last 108.65) - open/close 16445 (time 1777421981.81 total 166.67 last 118.26) - open/close 17608 (time 1777421991.82 total 176.67 last 116.27) - open/close 18822 (time 1777422001.82 total 186.67 last 121.39) - open/close 19892 (time 1777422011.82 total 196.68 last 106.98) - open/close 20882 (time 1777422021.82 total 206.68 last 98.97) - open/close 21798 (time 1777422031.83 total 216.69 last 91.49) - open/close 22610 (time 1777422041.84 total 226.70 last 81.12) - open/close 23461 (time 1777422051.85 total 236.70 last 85.08) - open/close 24262 (time 1777422061.85 total 246.71 last 80.06) - open/close 25092 (time 1777422071.86 total 256.72 last 82.93) - open/close 26011 (time 1777422081.86 total 266.72 last 91.90) - open/close 27266 (time 1777422091.86 total 276.72 last 125.45) - open/close 28652 (time 1777422101.86 total 286.72 last 138.57) - open/close 30000 (time 1777422110.70 total 295.56 last 152.54) - open/close 31460 (time 1777422120.70 total 305.56 last 145.97) - open/close 33023 (time 1777422130.71 total 315.57 last 156.24) - open/close 35429 (time 1777422140.71 total 325.57 last 240.60) - open/close 37915 (time 1777422150.71 total 335.57 last 248.57) - open/close 40000 (time 1777422158.56 total 343.42 last 265.45) - open/close 42655 (time 1777422168.57 total 353.42 last 265.45) - open/close 44752 (time 1777422178.57 total 363.42 last 209.68) - open/close 46687 (time 1777422188.57 total 373.43 last 193.45) - open/close 48548 (time 1777422198.57 total 383.43 last 186.02) - open/close 50000 (time 1777422205.96 total 390.81 last 196.63) - open/close 51901 (time 1777422215.96 total 400.82 last 190.03) - open/close 53932 (time 1777422225.96 total 410.82 last 203.09) - open/close 55959 (time 1777422235.96 total 420.82 last 202.65) - open/close 58043 (time 1777422245.97 total 430.82 last 208.36) - open/close 59898 (time 1777422255.97 total 440.83 last 185.45) - open/close 62496 (time 1777422265.97 total 450.83 last 259.78) total: 64768 open/close in 460.64 seconds: 140.60 ops/second - /unlink 4970 (time 1777422287.54 total 10.00 last 496.99) - /unlink 9130 (time 1777422297.54 total 20.00 last 415.91) - /unlink 10000 (time 1777422299.61 total 22.07 last 420.62) - /unlink 15278 (time 1777422309.61 total 32.07 last 527.71) - /unlink 20000 (time 1777422317.58 total 40.05 last 592.27) - /unlink 25724 (time 1777422327.58 total 50.05 last 572.39) - /unlink 30000 (time 1777422336.76 total 59.22 last 465.97) - /unlink 35827 (time 1777422346.76 total 69.22 last 582.68) - /unlink 40000 (time 1777422353.77 total 76.23 last 595.33) - /unlink 45802 (time 1777422363.77 total 86.23 last 580.16) - /unlink 50000 (time 1777422372.15 total 94.61 last 500.81) - /unlink 56864 (time 1777422382.15 total 104.62 last 686.37) - /unlink 60000 (time 1777422387.29 total 109.75 last 610.41) total: 64768 /unlink in 117.06 seconds: 553.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3729 (time 1777422428.31 total 10.00 last 372.81) - open/close 7651 (time 1777422438.31 total 20.00 last 392.14) - open/close 10000 (time 1777422445.15 total 26.84 last 343.70) - open/close 13822 (time 1777422455.15 total 36.84 last 382.11) - open/close 16376 (time 1777422465.16 total 46.85 last 255.18) - open/close 19213 (time 1777422475.16 total 56.85 last 283.67) - open/close 20000 (time 1777422477.20 total 58.89 last 386.28) - open/close 23272 (time 1777422487.20 total 68.89 last 327.18) - open/close 26575 (time 1777422497.20 total 78.89 last 330.27) - open/close 29853 (time 1777422507.20 total 88.89 last 327.76) - open/close 33492 (time 1777422517.20 total 98.89 last 363.78) - open/close 35814 (time 1777422527.20 total 108.90 last 232.18) - open/close 38039 (time 1777422537.21 total 118.90 last 222.42) - open/close 40000 (time 1777422543.49 total 125.18 last 312.09) - open/close 43250 (time 1777422553.49 total 135.18 last 325.00) - open/close 45990 (time 1777422563.49 total 145.18 last 273.99) - open/close 49084 (time 1777422573.59 total 155.28 last 306.37) - open/close 50000 (time 1777422576.61 total 158.30 last 303.48) - open/close 53015 (time 1777422586.61 total 168.30 last 301.49) - open/close 56153 (time 1777422596.61 total 178.30 last 313.69) - open/close 58936 (time 1777422606.62 total 188.31 last 278.22) - open/close 60000 (time 1777422611.13 total 192.82 last 235.92) - open/close 62256 (time 1777422621.14 total 202.83 last 225.31) - open/close 64065 (time 1777422631.15 total 212.84 last 180.76) total: 64768 open/close in 216.92 seconds: 298.58 ops/second - /unlink 3501 (time 1777422647.09 total 10.00 last 350.03) - /unlink 6895 (time 1777422657.09 total 20.01 last 339.29) - /unlink 10000 (time 1777422664.86 total 27.77 last 399.80) - /unlink 13243 (time 1777422674.86 total 37.78 last 324.14) - /unlink 17760 (time 1777422684.87 total 47.78 last 451.63) - /unlink 20000 (time 1777422689.05 total 51.96 last 535.25) - /unlink 24498 (time 1777422699.05 total 61.96 last 449.75) - /unlink 28885 (time 1777422709.11 total 72.02 last 436.29) - /unlink 30000 (time 1777422711.98 total 74.90 last 387.41) - /unlink 34942 (time 1777422721.99 total 84.90 last 494.16) - /unlink 40000 (time 1777422730.23 total 93.14 last 613.82) - /unlink 45111 (time 1777422740.23 total 103.14 last 511.08) - /unlink 50000 (time 1777422749.54 total 112.45 last 525.05) - /unlink 55794 (time 1777422759.54 total 122.45 last 579.33) - /unlink 60000 (time 1777422766.12 total 129.04 last 638.85) total: 64768 /unlink in 137.75 seconds: 470.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3739 (time 1777422801.57 total 10.00 last 373.88) - open/close 8107 (time 1777422811.57 total 20.00 last 436.77) - open/close 10000 (time 1777422816.27 total 24.70 last 402.51) - open/close 13467 (time 1777422826.27 total 34.70 last 346.68) - open/close 17514 (time 1777422836.28 total 44.71 last 404.57) - open/close 20000 (time 1777422842.66 total 51.10 last 389.22) - open/close 24218 (time 1777422852.66 total 61.10 last 421.80) - open/close 28111 (time 1777422862.66 total 71.10 last 389.22) - open/close 30000 (time 1777422867.05 total 75.48 last 430.72) - open/close 33985 (time 1777422877.05 total 85.49 last 398.43) - open/close 38286 (time 1777422887.05 total 95.49 last 430.09) - open/close 40000 (time 1777422890.75 total 99.18 last 463.99) - open/close 44610 (time 1777422900.75 total 109.18 last 461.00) - open/close 49143 (time 1777422910.81 total 119.25 last 450.29) - open/close 53639 (time 1777422920.82 total 129.25 last 449.52) - open/close 57649 (time 1777422930.84 total 139.27 last 400.13) - open/close 60000 (time 1777422936.82 total 145.25 last 393.11) - open/close 63811 (time 1777422946.82 total 155.25 last 381.04) total: 64768 open/close in 157.46 seconds: 411.33 ops/second - /unlink 6920 (time 1777422960.36 total 10.00 last 691.92) - /unlink 10000 (time 1777422965.29 total 14.92 last 625.60) - /unlink 16801 (time 1777422975.29 total 24.93 last 680.04) - /unlink 20000 (time 1777422980.09 total 29.73 last 666.04) - /unlink 25358 (time 1777422990.09 total 39.73 last 535.76) - /unlink 30000 (time 1777422997.18 total 46.82 last 654.86) - /unlink 35913 (time 1777423007.18 total 56.82 last 591.25) - /unlink 40000 (time 1777423014.26 total 63.90 last 577.21) - /unlink 47979 (time 1777423024.26 total 73.90 last 797.78) - /unlink 50000 (time 1777423026.69 total 76.32 last 833.85) - /unlink 57451 (time 1777423036.69 total 86.33 last 745.00) - /unlink 60000 (time 1777423040.21 total 89.84 last 725.02) total: 64768 /unlink in 96.43 seconds: 671.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4654 (time 1777423078.65 total 10.00 last 465.35) - open/close 9490 (time 1777423088.65 total 20.00 last 483.53) - open/close 13658 (time 1777423098.65 total 30.00 last 416.77) - open/close 18035 (time 1777423108.65 total 40.00 last 437.64) - open/close 20000 (time 1777423114.93 total 46.28 last 312.99) - open/close 24162 (time 1777423124.93 total 56.28 last 416.15) - open/close 28879 (time 1777423134.93 total 66.28 last 471.66) - open/close 30000 (time 1777423137.12 total 68.48 last 511.39) - open/close 34663 (time 1777423147.12 total 78.48 last 466.30) - open/close 39334 (time 1777423157.13 total 88.48 last 466.93) - open/close 43661 (time 1777423167.13 total 98.48 last 432.70) - open/close 48696 (time 1777423177.13 total 108.48 last 503.47) - open/close 50000 (time 1777423179.60 total 110.96 last 526.84) - open/close 54235 (time 1777423189.63 total 120.98 last 422.41) - open/close 58500 (time 1777423199.63 total 130.98 last 426.42) - open/close 60000 (time 1777423204.02 total 135.37 last 342.08) - open/close 61349 (time 1777423214.02 total 145.38 last 134.80) - open/close 62413 (time 1777423224.04 total 155.39 last 106.22) - open/close 63183 (time 1777423234.04 total 165.39 last 76.99) - open/close 63926 (time 1777423244.06 total 175.41 last 74.18) total: 64768 open/close in 185.14 seconds: 349.83 ops/second - /unlink 2872 (time 1777423265.94 total 10.00 last 287.09) - /unlink 5827 (time 1777423275.94 total 20.01 last 295.45) - /unlink 9097 (time 1777423285.94 total 30.01 last 326.95) - /unlink 10000 (time 1777423289.47 total 33.54 last 255.77) - /unlink 12648 (time 1777423299.47 total 43.54 last 264.70) - /unlink 15045 (time 1777423309.48 total 53.54 last 239.64) - /unlink 17586 (time 1777423319.48 total 63.55 last 254.02) - /unlink 20000 (time 1777423328.71 total 72.78 last 261.49) - /unlink 22448 (time 1777423338.71 total 82.78 last 244.75) - /unlink 25308 (time 1777423348.71 total 92.78 last 285.96) - /unlink 27706 (time 1777423358.72 total 102.78 last 239.75) - /unlink 30000 (time 1777423365.95 total 110.01 last 317.30) - /unlink 32612 (time 1777423375.95 total 120.01 last 261.20) - /unlink 35408 (time 1777423385.95 total 130.02 last 279.55) - /unlink 37730 (time 1777423395.95 total 140.02 last 232.15) - /unlink 40000 (time 1777423404.43 total 148.50 last 267.67) - /unlink 42419 (time 1777423414.43 total 158.50 last 241.89) - /unlink 44843 (time 1777423424.43 total 168.50 last 242.39) - /unlink 46980 (time 1777423434.44 total 178.50 last 213.62) - /unlink 49135 (time 1777423444.44 total 188.50 last 215.49) - /unlink 50000 (time 1777423448.28 total 192.35 last 225.05) - /unlink 52307 (time 1777423458.28 total 202.35 last 230.67) - /unlink 54623 (time 1777423468.28 total 212.35 last 231.54) - /unlink 56609 (time 1777423478.29 total 222.36 last 198.45) - /unlink 58960 (time 1777423488.29 total 232.36 last 235.09) - /unlink 60000 (time 1777423492.72 total 236.79 last 234.67) - /unlink 62791 (time 1777423502.73 total 246.79 last 279.03) total: 64768 /unlink in 254.64 seconds: 254.35 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2