== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:18:46 (1752509926) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) oleg335-server: oleg335-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg335-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1300 (time 1752509984.65 total 10.00 last 129.95) - open/close 2674 (time 1752509994.65 total 20.01 last 137.36) - open/close 4577 (time 1752510004.66 total 30.01 last 190.18) - open/close 6700 (time 1752510014.66 total 40.02 last 212.22) - open/close 8554 (time 1752510024.66 total 50.02 last 185.35) - open/close 9913 (time 1752510034.67 total 60.03 last 135.79) - open/close 11526 (time 1752510044.67 total 70.03 last 161.28) - open/close 13041 (time 1752510054.67 total 80.03 last 151.50) - open/close 14343 (time 1752510064.67 total 90.03 last 130.18) - open/close 15777 (time 1752510074.68 total 100.04 last 143.29) - open/close 17438 (time 1752510084.69 total 110.04 last 166.00) - open/close 19132 (time 1752510094.69 total 120.05 last 169.36) - open/close 20000 (time 1752510098.93 total 124.28 last 204.87) - open/close 21778 (time 1752510108.93 total 134.29 last 177.73) - open/close 23673 (time 1752510118.93 total 144.29 last 189.48) - open/close 25578 (time 1752510128.93 total 154.29 last 190.47) - open/close 27674 (time 1752510138.94 total 164.29 last 209.55) - open/close 29736 (time 1752510148.94 total 174.29 last 206.15) - open/close 31530 (time 1752510158.94 total 184.30 last 179.38) - open/close 33472 (time 1752510168.94 total 194.30 last 194.16) - open/close 35616 (time 1752510178.95 total 204.30 last 214.34) - open/close 37622 (time 1752510188.95 total 214.30 last 200.57) - open/close 39614 (time 1752510198.95 total 224.30 last 199.18) - open/close 41460 (time 1752510208.95 total 234.30 last 184.58) - open/close 43245 (time 1752510218.95 total 244.31 last 178.43) - open/close 45090 (time 1752510228.96 total 254.31 last 184.45) - open/close 47161 (time 1752510238.96 total 264.32 last 206.99) - open/close 49052 (time 1752510248.96 total 274.32 last 189.06) - open/close 50000 (time 1752510254.10 total 279.45 last 184.72) - open/close 52003 (time 1752510264.10 total 289.46 last 200.18) - open/close 53875 (time 1752510274.11 total 299.46 last 187.12) - open/close 55607 (time 1752510284.11 total 309.47 last 173.06) - open/close 57630 (time 1752510294.12 total 319.47 last 202.26) - open/close 59588 (time 1752510304.12 total 329.47 last 195.80) - open/close 60000 (time 1752510306.23 total 331.59 last 194.64) - open/close 62227 (time 1752510316.24 total 341.59 last 222.64) - open/close 64378 (time 1752510326.24 total 351.59 last 215.07) total: 64768 open/close in 353.48 seconds: 183.23 ops/second - /unlink 3955 (time 1752510339.82 total 10.00 last 395.42) - /unlink 7766 (time 1752510349.82 total 20.00 last 381.09) - /unlink 10000 (time 1752510356.95 total 27.13 last 313.55) - /unlink 13389 (time 1752510366.95 total 37.13 last 338.72) - /unlink 16709 (time 1752510376.95 total 47.13 last 331.94) - /unlink 19557 (time 1752510386.96 total 57.14 last 284.65) - /unlink 23016 (time 1752510396.96 total 67.14 last 345.89) - /unlink 26417 (time 1752510406.96 total 77.14 last 340.01) - /unlink 29846 (time 1752510416.97 total 87.14 last 342.81) - /unlink 33673 (time 1752510426.97 total 97.15 last 382.68) - /unlink 37465 (time 1752510436.97 total 107.15 last 379.15) - /unlink 40000 (time 1752510443.42 total 113.60 last 392.96) - /unlink 44046 (time 1752510453.43 total 123.61 last 404.09) - /unlink 47534 (time 1752510463.43 total 133.61 last 348.73) - /unlink 50000 (time 1752510470.52 total 140.69 last 348.21) - /unlink 53867 (time 1752510480.52 total 150.69 last 386.69) - /unlink 57848 (time 1752510490.52 total 160.70 last 398.03) - /unlink 60000 (time 1752510495.99 total 166.16 last 393.55) - /unlink 63812 (time 1752510505.99 total 176.17 last 381.15) total: 64768 /unlink in 179.04 seconds: 361.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2173 (time 1752510539.30 total 10.00 last 217.24) - open/close 4572 (time 1752510549.30 total 20.00 last 239.85) - open/close 7082 (time 1752510559.30 total 30.01 last 250.99) - open/close 10000 (time 1752510567.96 total 38.67 last 336.87) - open/close 12747 (time 1752510577.96 total 48.67 last 274.68) - open/close 15410 (time 1752510587.96 total 58.67 last 266.28) - open/close 18271 (time 1752510597.97 total 68.67 last 286.02) - open/close 20000 (time 1752510605.09 total 75.80 last 242.56) - open/close 23188 (time 1752510615.10 total 85.80 last 318.73) - open/close 26487 (time 1752510625.10 total 95.80 last 329.85) - open/close 29427 (time 1752510635.10 total 105.80 last 293.94) - open/close 32557 (time 1752510645.10 total 115.81 last 312.99) - open/close 35666 (time 1752510655.11 total 125.82 last 310.47) - open/close 38251 (time 1752510665.12 total 135.82 last 258.43) - open/close 40000 (time 1752510671.14 total 141.84 last 290.56) - open/close 42681 (time 1752510681.14 total 151.84 last 268.04) - open/close 45750 (time 1752510691.14 total 161.85 last 306.84) - open/close 48417 (time 1752510701.14 total 171.85 last 266.60) - open/close 50000 (time 1752510707.16 total 177.86 last 263.19) - open/close 52238 (time 1752510717.16 total 187.87 last 223.70) - open/close 54825 (time 1752510727.16 total 197.87 last 258.67) - open/close 57199 (time 1752510737.17 total 207.87 last 237.32) - open/close 59485 (time 1752510747.17 total 217.88 last 228.51) - open/close 60000 (time 1752510749.52 total 220.22 last 219.78) - open/close 62037 (time 1752510759.52 total 230.22 last 203.61) - open/close 64142 (time 1752510769.52 total 240.23 last 210.42) total: 64768 open/close in 242.92 seconds: 266.62 ops/second - /unlink 4478 (time 1752510783.89 total 10.00 last 447.74) - /unlink 8633 (time 1752510793.89 total 20.00 last 415.39) - /unlink 10000 (time 1752510796.69 total 22.80 last 488.36) - /unlink 13790 (time 1752510806.74 total 32.85 last 377.09) - /unlink 17523 (time 1752510816.74 total 42.86 last 373.25) - /unlink 20000 (time 1752510823.78 total 49.89 last 352.19) - /unlink 23843 (time 1752510833.78 total 59.89 last 384.25) - /unlink 27817 (time 1752510843.78 total 69.89 last 397.34) - /unlink 30000 (time 1752510848.45 total 74.57 last 467.07) - /unlink 34484 (time 1752510858.45 total 84.57 last 448.32) - /unlink 39388 (time 1752510868.46 total 94.57 last 490.35) - /unlink 44470 (time 1752510878.46 total 104.57 last 508.18) - /unlink 49935 (time 1752510888.46 total 114.57 last 546.44) - /unlink 55256 (time 1752510898.46 total 124.57 last 532.07) - /unlink 59476 (time 1752510908.46 total 134.57 last 421.99) - /unlink 62967 (time 1752510918.46 total 144.57 last 349.05) total: 64768 /unlink in 150.13 seconds: 431.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2658 (time 1752510957.07 total 10.00 last 265.74) - open/close 5716 (time 1752510967.07 total 20.00 last 305.80) - open/close 8795 (time 1752510977.07 total 30.01 last 307.81) - open/close 10000 (time 1752510980.80 total 33.73 last 323.18) - open/close 12622 (time 1752510990.80 total 43.73 last 262.18) - open/close 15343 (time 1752511000.80 total 53.74 last 271.91) - open/close 17887 (time 1752511010.81 total 63.74 last 254.39) - open/close 20000 (time 1752511020.25 total 73.19 last 223.69) - open/close 23334 (time 1752511030.25 total 83.19 last 333.31) - open/close 25742 (time 1752511040.26 total 93.19 last 240.74) - open/close 28509 (time 1752511050.26 total 103.19 last 276.69) - open/close 30000 (time 1752511055.32 total 108.26 last 294.55) - open/close 32880 (time 1752511065.32 total 118.26 last 287.92) - open/close 35327 (time 1752511075.32 total 128.26 last 244.66) - open/close 38815 (time 1752511085.32 total 138.26 last 348.79) - open/close 40000 (time 1752511088.38 total 141.32 last 387.14) - open/close 43524 (time 1752511098.38 total 151.32 last 352.38) - open/close 46632 (time 1752511108.39 total 161.32 last 310.77) - open/close 49717 (time 1752511118.39 total 171.32 last 308.48) - open/close 52231 (time 1752511128.39 total 181.32 last 251.39) - open/close 54934 (time 1752511138.39 total 191.32 last 270.29) - open/close 58197 (time 1752511148.39 total 201.33 last 326.24) - open/close 60000 (time 1752511154.18 total 207.11 last 311.52) - open/close 63464 (time 1752511164.18 total 217.11 last 346.36) total: 64768 open/close in 220.75 seconds: 293.40 ops/second - /unlink 6957 (time 1752511179.17 total 10.00 last 695.64) - /unlink 10000 (time 1752511184.46 total 15.29 last 575.26) - /unlink 15695 (time 1752511194.47 total 25.29 last 569.43) - /unlink 20000 (time 1752511200.78 total 31.61 last 681.67) - /unlink 26974 (time 1752511210.78 total 41.61 last 697.36) - /unlink 30000 (time 1752511214.67 total 45.49 last 778.66) - /unlink 37014 (time 1752511224.67 total 55.49 last 701.34) - /unlink 40000 (time 1752511228.26 total 59.09 last 831.48) - /unlink 47299 (time 1752511238.26 total 69.09 last 729.85) - /unlink 50000 (time 1752511242.90 total 73.73 last 581.55) - /unlink 56363 (time 1752511252.91 total 83.73 last 636.20) - /unlink 60000 (time 1752511258.65 total 89.47 last 633.52) total: 64768 /unlink in 97.15 seconds: 666.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5161 (time 1752511296.30 total 10.00 last 515.98) - open/close 9642 (time 1752511306.30 total 20.00 last 448.10) - open/close 14433 (time 1752511316.30 total 30.00 last 479.02) - open/close 18615 (time 1752511326.30 total 40.00 last 418.17) - open/close 20000 (time 1752511329.91 total 43.61 last 384.47) - open/close 23869 (time 1752511339.91 total 53.61 last 386.80) - open/close 27475 (time 1752511349.91 total 63.61 last 360.47) - open/close 30000 (time 1752511357.97 total 71.67 last 313.56) - open/close 34105 (time 1752511367.97 total 81.67 last 410.41) - open/close 37703 (time 1752511377.97 total 91.67 last 359.72) - open/close 40000 (time 1752511384.32 total 98.02 last 361.89) - open/close 43254 (time 1752511394.32 total 108.02 last 325.32) - open/close 44411 (time 1752511404.33 total 118.03 last 115.63) - open/close 45306 (time 1752511414.34 total 128.04 last 89.35) - open/close 46144 (time 1752511424.35 total 138.05 last 83.72) - open/close 47089 (time 1752511434.35 total 148.05 last 94.48) - open/close 48052 (time 1752511444.36 total 158.06 last 96.28) - open/close 49246 (time 1752511454.36 total 168.06 last 119.38) - open/close 50000 (time 1752511460.64 total 174.35 last 119.92) - open/close 51472 (time 1752511470.65 total 184.35 last 147.15) - open/close 52783 (time 1752511480.66 total 194.36 last 130.92) - open/close 54171 (time 1752511490.66 total 204.36 last 138.79) - open/close 55677 (time 1752511500.66 total 214.36 last 150.59) - open/close 57814 (time 1752511510.66 total 224.36 last 213.69) - open/close 59654 (time 1752511520.67 total 234.37 last 183.94) - open/close 60000 (time 1752511523.17 total 236.87 last 138.43) - open/close 62099 (time 1752511533.17 total 246.87 last 209.82) - open/close 63630 (time 1752511543.17 total 256.87 last 153.07) total: 64768 open/close in 264.04 seconds: 245.30 ops/second - /unlink 2697 (time 1752511562.24 total 10.00 last 269.63) - /unlink 6221 (time 1752511572.24 total 20.00 last 352.34) - /unlink 9722 (time 1752511582.24 total 30.01 last 350.04) - /unlink 11942 (time 1752511592.25 total 40.01 last 221.93) - /unlink 14234 (time 1752511602.25 total 50.01 last 229.15) - /unlink 16857 (time 1752511612.25 total 60.01 last 262.29) - /unlink 19718 (time 1752511622.25 total 70.01 last 286.08) - /unlink 21952 (time 1752511632.25 total 80.02 last 223.31) - /unlink 24974 (time 1752511642.25 total 90.02 last 302.16) - /unlink 27864 (time 1752511652.26 total 100.02 last 288.86) - /unlink 30000 (time 1752511660.70 total 108.46 last 253.15) - /unlink 32965 (time 1752511670.70 total 118.46 last 296.44) - /unlink 35587 (time 1752511680.70 total 128.46 last 262.19) - /unlink 39357 (time 1752511690.70 total 138.47 last 376.89) - /unlink 43290 (time 1752511700.70 total 148.47 last 393.29) - /unlink 47217 (time 1752511710.70 total 158.47 last 392.67) - /unlink 50000 (time 1752511718.94 total 166.70 last 337.90) - /unlink 53523 (time 1752511728.94 total 176.70 last 352.23) - /unlink 57615 (time 1752511738.94 total 186.71 last 409.12) - /unlink 60000 (time 1752511745.92 total 193.68 last 341.86) - /unlink 63175 (time 1752511755.92 total 203.69 last 317.42) total: 64768 /unlink in 209.29 seconds: 309.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1249 (time 1752511792.17 total 10.00 last 124.86) - open/close 2435 (time 1752511802.18 total 20.01 last 118.51) - open/close 3730 (time 1752511812.19 total 30.01 last 129.46) - open/close 4959 (time 1752511822.19 total 40.02 last 122.88) - open/close 6881 (time 1752511832.19 total 50.02 last 192.16) - open/close 8586 (time 1752511842.19 total 60.02 last 170.47) - open/close 10000 (time 1752511849.85 total 67.67 last 184.72) - open/close 11662 (time 1752511859.85 total 77.68 last 166.15) - open/close 13715 (time 1752511869.85 total 87.68 last 205.20) - open/close 15927 (time 1752511879.86 total 97.69 last 221.08) - open/close 18128 (time 1752511889.86 total 107.69 last 220.09) - open/close 20000 (time 1752511897.76 total 115.59 last 236.91) - open/close 22155 (time 1752511907.76 total 125.59 last 215.49) - open/close 25305 (time 1752511917.78 total 135.61 last 314.43) - open/close 27854 (time 1752511927.78 total 145.61 last 254.88) - open/close 30000 (time 1752511935.25 total 153.08 last 287.36) - open/close 32413 (time 1752511945.25 total 163.08 last 241.22) - open/close 34504 (time 1752511955.25 total 173.08 last 209.04) - open/close 37107 (time 1752511965.26 total 183.09 last 260.26) - open/close 39491 (time 1752511975.26 total 193.09 last 238.39) - open/close 40000 (time 1752511978.22 total 196.05 last 171.70) - open/close 41718 (time 1752511988.22 total 206.05 last 171.76) - open/close 43368 (time 1752511998.23 total 216.06 last 164.94) - open/close 44957 (time 1752512008.23 total 226.06 last 158.82) - open/close 46459 (time 1752512018.24 total 236.07 last 150.12) - open/close 48232 (time 1752512028.24 total 246.07 last 177.18) - open/close 50000 (time 1752512037.74 total 255.57 last 186.21) - open/close 51872 (time 1752512047.75 total 265.57 last 187.07) - open/close 53590 (time 1752512057.75 total 275.58 last 171.79) - open/close 54985 (time 1752512067.75 total 285.58 last 139.48) - open/close 56396 (time 1752512077.75 total 295.58 last 141.04) - open/close 57788 (time 1752512087.76 total 305.59 last 139.08) - open/close 59401 (time 1752512097.76 total 315.59 last 161.29) - open/close 60000 (time 1752512101.60 total 319.42 last 156.26) - open/close 61471 (time 1752512111.60 total 329.43 last 147.06) - open/close 63030 (time 1752512121.60 total 339.43 last 155.83) total: 64768 open/close in 348.88 seconds: 185.64 ops/second - /unlink 3214 (time 1752512142.86 total 10.00 last 321.30) - /unlink 6115 (time 1752512152.86 total 20.01 last 290.02) - /unlink 9408 (time 1752512162.87 total 30.01 last 329.17) - /unlink 10000 (time 1752512165.16 total 32.31 last 257.82) - /unlink 12476 (time 1752512175.16 total 42.31 last 247.55) - /unlink 15717 (time 1752512185.17 total 52.31 last 324.02) - /unlink 19004 (time 1752512195.17 total 62.31 last 328.66) - /unlink 20000 (time 1752512197.93 total 65.07 last 360.51) - /unlink 22972 (time 1752512207.93 total 75.08 last 297.16) - /unlink 26515 (time 1752512217.93 total 85.08 last 354.26) - /unlink 30000 (time 1752512227.65 total 94.79 last 358.66) - /unlink 32950 (time 1752512237.65 total 104.80 last 294.93) - /unlink 35559 (time 1752512247.66 total 114.80 last 260.79) - /unlink 38325 (time 1752512257.66 total 124.80 last 276.53) - /unlink 40000 (time 1752512263.46 total 130.61 last 288.65) - /unlink 43280 (time 1752512273.47 total 140.61 last 327.89) - /unlink 46977 (time 1752512283.47 total 150.61 last 369.64) - /unlink 50000 (time 1752512289.27 total 156.41 last 521.13) - /unlink 55247 (time 1752512299.27 total 166.41 last 524.56) - /unlink 59475 (time 1752512309.27 total 176.41 last 422.78) - /unlink 64196 (time 1752512319.27 total 186.42 last 472.04) total: 64768 /unlink in 187.41 seconds: 345.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2