== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:19:14 (1752509954) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) oleg245-server: oleg245-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg245-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg245-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1177 (time 1752510023.50 total 10.00 last 117.64) - open/close 2671 (time 1752510033.51 total 20.01 last 149.32) - open/close 4320 (time 1752510043.51 total 30.01 last 164.88) - open/close 6822 (time 1752510053.51 total 40.01 last 250.19) - open/close 8964 (time 1752510063.51 total 50.02 last 214.12) - open/close 10000 (time 1752510068.59 total 55.09 last 204.05) - open/close 12248 (time 1752510078.59 total 65.10 last 224.74) - open/close 13957 (time 1752510088.60 total 75.10 last 170.82) - open/close 16059 (time 1752510098.60 total 85.10 last 210.14) - open/close 18031 (time 1752510108.60 total 95.10 last 197.17) - open/close 20000 (time 1752510116.80 total 103.31 last 240.06) - open/close 22193 (time 1752510126.81 total 113.31 last 219.23) - open/close 25214 (time 1752510136.81 total 123.31 last 302.02) - open/close 27648 (time 1752510146.81 total 133.31 last 243.39) - open/close 29455 (time 1752510156.81 total 143.31 last 180.67) - open/close 30000 (time 1752510159.66 total 146.17 last 191.18) - open/close 32128 (time 1752510169.67 total 156.17 last 212.70) - open/close 34423 (time 1752510179.67 total 166.17 last 229.45) - open/close 36442 (time 1752510189.67 total 176.17 last 201.85) - open/close 38981 (time 1752510199.67 total 186.18 last 253.81) - open/close 40000 (time 1752510204.00 total 190.50 last 235.70) - open/close 42477 (time 1752510214.00 total 200.50 last 247.64) - open/close 44845 (time 1752510224.00 total 210.51 last 236.72) - open/close 47137 (time 1752510234.01 total 220.51 last 229.14) - open/close 49479 (time 1752510244.01 total 230.51 last 234.12) - open/close 50000 (time 1752510246.12 total 232.63 last 246.27) - open/close 52317 (time 1752510256.13 total 242.63 last 231.66) - open/close 54583 (time 1752510266.13 total 252.63 last 226.51) - open/close 57328 (time 1752510276.13 total 262.64 last 274.48) - open/close 60000 (time 1752510284.58 total 271.09 last 316.07) - open/close 62745 (time 1752510294.59 total 281.10 last 274.29) total: 64768 open/close in 289.31 seconds: 223.87 ops/second - /unlink 5078 (time 1752510314.47 total 10.00 last 507.70) - /unlink 10000 (time 1752510322.52 total 18.05 last 611.95) - /unlink 16255 (time 1752510332.52 total 28.05 last 625.42) - /unlink 20000 (time 1752510340.21 total 35.74 last 486.89) - /unlink 24776 (time 1752510350.21 total 45.74 last 477.49) - /unlink 29498 (time 1752510360.21 total 55.74 last 472.15) - /unlink 34418 (time 1752510370.21 total 65.74 last 492.00) - /unlink 40000 (time 1752510379.60 total 75.13 last 594.71) - /unlink 45602 (time 1752510389.60 total 85.13 last 560.17) - /unlink 50000 (time 1752510396.18 total 91.71 last 667.89) - /unlink 56293 (time 1752510406.19 total 101.71 last 629.22) - /unlink 60000 (time 1752510411.72 total 107.25 last 670.08) total: 64768 /unlink in 115.04 seconds: 563.00 ops/second Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 - open/close 3125 (time 1752510441.30 total 10.00 last 312.49) - open/close 7209 (time 1752510451.30 total 20.00 last 408.26) - open/close 10000 (time 1752510460.09 total 28.79 last 317.59) - open/close 13633 (time 1752510470.09 total 38.79 last 363.27) - open/close 17370 (time 1752510480.09 total 48.79 last 373.69) - open/close 20000 (time 1752510486.28 total 54.99 last 424.55) - open/close 24249 (time 1752510496.29 total 64.99 last 424.84) - open/close 27699 (time 1752510506.29 total 74.99 last 344.93) - open/close 30000 (time 1752510512.75 total 81.46 last 355.84) - open/close 33129 (time 1752510522.75 total 91.46 last 312.87) - open/close 36572 (time 1752510532.76 total 101.46 last 344.27) - open/close 40000 (time 1752510542.75 total 111.45 last 342.97) - open/close 43039 (time 1752510552.75 total 121.46 last 303.87) - open/close 46969 (time 1752510562.75 total 131.46 last 392.90) - open/close 50000 (time 1752510571.87 total 140.58 last 332.43) - open/close 54381 (time 1752510581.87 total 150.58 last 438.06) - open/close 57647 (time 1752510591.87 total 160.58 last 326.59) - open/close 59996 (time 1752510601.88 total 170.58 last 234.85) - open/close 62804 (time 1752510611.88 total 180.58 last 280.76) total: 64768 open/close in 187.12 seconds: 346.13 ops/second - /unlink 4132 (time 1752510630.03 total 10.00 last 413.16) - /unlink 8648 (time 1752510640.03 total 20.00 last 451.52) - /unlink 10000 (time 1752510643.28 total 23.25 last 416.49) - /unlink 14509 (time 1752510653.28 total 33.25 last 450.86) - /unlink 19915 (time 1752510663.28 total 43.25 last 540.59) - /unlink 25556 (time 1752510673.28 total 53.25 last 564.08) - /unlink 30000 (time 1752510681.24 total 61.21 last 558.51) - /unlink 37208 (time 1752510691.24 total 71.21 last 720.79) - /unlink 40000 (time 1752510696.18 total 76.15 last 564.80) - /unlink 45882 (time 1752510706.18 total 86.15 last 588.18) - /unlink 49975 (time 1752510716.18 total 96.15 last 409.20) - /unlink 54653 (time 1752510726.18 total 106.16 last 467.72) - /unlink 60000 (time 1752510735.26 total 115.24 last 588.85) total: 64768 /unlink in 121.58 seconds: 532.72 ops/second Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 - open/close 4944 (time 1752510778.76 total 10.00 last 494.33) - open/close 9345 (time 1752510788.76 total 20.00 last 440.02) - open/close 13720 (time 1752510798.77 total 30.01 last 437.16) - open/close 17959 (time 1752510808.77 total 40.01 last 423.81) - open/close 20000 (time 1752510812.93 total 44.17 last 490.74) - open/close 24755 (time 1752510822.93 total 54.17 last 475.43) - open/close 29315 (time 1752510832.93 total 64.18 last 455.90) - open/close 33212 (time 1752510842.94 total 74.18 last 389.58) - open/close 37426 (time 1752510852.94 total 84.18 last 421.34) - open/close 40000 (time 1752510858.10 total 89.34 last 498.82) - open/close 44701 (time 1752510868.10 total 99.34 last 469.99) - open/close 49592 (time 1752510878.10 total 109.34 last 489.06) - open/close 54661 (time 1752510888.10 total 119.34 last 506.84) - open/close 59685 (time 1752510898.10 total 129.35 last 502.32) - open/close 64739 (time 1752510908.10 total 139.35 last 505.39) total: 64768 open/close in 139.40 seconds: 464.62 ops/second - /unlink 8619 (time 1752510919.45 total 10.00 last 861.81) - /unlink 17530 (time 1752510929.45 total 20.00 last 891.03) - /unlink 20000 (time 1752510932.07 total 22.62 last 944.25) - /unlink 28603 (time 1752510942.07 total 32.62 last 860.19) - /unlink 36224 (time 1752510952.07 total 42.62 last 762.04) - /unlink 40000 (time 1752510957.21 total 47.76 last 735.24) - /unlink 47365 (time 1752510967.21 total 57.76 last 736.48) - /unlink 50000 (time 1752510970.55 total 61.10 last 788.20) - /unlink 57636 (time 1752510980.55 total 71.10 last 763.53) - /unlink 60000 (time 1752510983.72 total 74.26 last 746.87) total: 64768 /unlink in 80.74 seconds: 802.21 ops/second Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 - open/close 4145 (time 1752511046.15 total 10.00 last 414.44) - open/close 7914 (time 1752511056.15 total 20.00 last 376.89) - open/close 10000 (time 1752511061.49 total 25.35 last 390.04) - open/close 13818 (time 1752511071.50 total 35.35 last 381.75) - open/close 18017 (time 1752511081.50 total 45.35 last 419.80) - open/close 20000 (time 1752511086.53 total 50.38 last 394.34) - open/close 24841 (time 1752511096.53 total 60.38 last 484.03) - open/close 29642 (time 1752511106.53 total 70.38 last 480.08) - open/close 33733 (time 1752511116.53 total 80.38 last 409.08) - open/close 38404 (time 1752511126.53 total 90.39 last 467.07) - open/close 40000 (time 1752511129.51 total 93.36 last 536.27) - open/close 45045 (time 1752511139.51 total 103.36 last 504.49) - open/close 50000 (time 1752511149.46 total 113.32 last 497.79) - open/close 54632 (time 1752511159.46 total 123.32 last 463.14) - open/close 58890 (time 1752511169.46 total 133.32 last 425.79) - open/close 60000 (time 1752511171.88 total 135.73 last 459.54) total: 64768 open/close in 145.66 seconds: 444.66 ops/second - /unlink 8321 (time 1752511193.19 total 10.03 last 829.79) - /unlink 10000 (time 1752511195.73 total 12.56 last 662.76) - /unlink 17995 (time 1752511205.73 total 22.56 last 799.43) - /unlink 20000 (time 1752511208.28 total 25.12 last 783.89) - /unlink 27843 (time 1752511218.28 total 35.12 last 784.25) - /unlink 30000 (time 1752511221.16 total 37.99 last 750.74) - /unlink 38102 (time 1752511231.16 total 47.99 last 810.12) - /unlink 40000 (time 1752511233.65 total 50.48 last 762.74) - /unlink 48402 (time 1752511243.65 total 60.48 last 840.15) - /unlink 56286 (time 1752511253.65 total 70.49 last 788.28) - /unlink 60000 (time 1752511257.85 total 74.69 last 883.63) total: 64768 /unlink in 79.76 seconds: 812.08 ops/second Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 - open/close 4028 (time 1752511348.23 total 10.00 last 402.77) - open/close 8320 (time 1752511358.23 total 20.00 last 429.18) - open/close 10000 (time 1752511362.54 total 24.31 last 390.33) - open/close 14552 (time 1752511372.54 total 34.31 last 455.20) - open/close 18466 (time 1752511382.54 total 44.31 last 391.36) - open/close 20000 (time 1752511387.10 total 48.86 last 336.54) - open/close 22897 (time 1752511397.10 total 58.87 last 289.66) - open/close 24270 (time 1752511407.10 total 68.87 last 137.24) - open/close 25313 (time 1752511417.11 total 78.88 last 104.22) - open/close 26279 (time 1752511427.12 total 88.89 last 96.51) - open/close 27307 (time 1752511437.12 total 98.89 last 102.75) - open/close 28742 (time 1752511447.13 total 108.90 last 143.43) - open/close 30000 (time 1752511456.40 total 118.16 last 135.74) - open/close 31516 (time 1752511466.40 total 128.16 last 151.60) - open/close 33298 (time 1752511476.40 total 138.17 last 178.12) - open/close 34856 (time 1752511486.40 total 148.17 last 155.75) - open/close 36334 (time 1752511496.41 total 158.17 last 147.78) - open/close 38129 (time 1752511506.41 total 168.17 last 179.49) - open/close 39714 (time 1752511516.41 total 178.17 last 158.49) - open/close 42523 (time 1752511526.41 total 188.18 last 280.87) - open/close 44612 (time 1752511536.41 total 198.18 last 208.90) - open/close 47041 (time 1752511546.41 total 208.18 last 242.83) - open/close 48454 (time 1752511556.41 total 218.18 last 141.29) - open/close 50000 (time 1752511566.18 total 227.95 last 158.30) - open/close 52262 (time 1752511576.18 total 237.95 last 226.19) - open/close 54568 (time 1752511586.18 total 247.95 last 230.58) - open/close 56453 (time 1752511596.18 total 257.95 last 188.42) - open/close 58761 (time 1752511606.18 total 267.95 last 230.79) - open/close 60000 (time 1752511610.25 total 272.02 last 304.39) - open/close 62699 (time 1752511620.26 total 282.02 last 269.83) total: 64768 open/close in 290.26 seconds: 223.14 ops/second - /unlink 5342 (time 1752511640.11 total 10.00 last 534.17) - /unlink 10000 (time 1752511647.61 total 17.50 last 620.80) - /unlink 17043 (time 1752511657.61 total 27.50 last 704.25) - /unlink 20000 (time 1752511661.50 total 31.39 last 761.21) - /unlink 27549 (time 1752511671.50 total 41.39 last 754.84) - /unlink 30000 (time 1752511674.78 total 44.67 last 746.67) - /unlink 37969 (time 1752511684.78 total 54.67 last 796.83) - /unlink 40000 (time 1752511687.32 total 57.21 last 800.77) - /unlink 47517 (time 1752511697.32 total 67.21 last 751.52) - /unlink 50000 (time 1752511701.24 total 71.13 last 633.31) - /unlink 55840 (time 1752511711.24 total 81.13 last 583.99) - /unlink 60000 (time 1752511718.56 total 88.45 last 568.40) total: 64768 /unlink in 96.66 seconds: 670.06 ops/second Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg245-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2