== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:45:02 (1772167502) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3911 (time 1772167559.72 total 10.00 last 391.03) - open/close 7642 (time 1772167569.72 total 20.00 last 373.06) - open/close 10000 (time 1772167576.02 total 26.31 last 374.05) - open/close 13686 (time 1772167586.02 total 36.31 last 368.51) - open/close 17316 (time 1772167596.03 total 46.31 last 362.95) - open/close 20000 (time 1772167603.66 total 53.94 last 351.74) - open/close 23336 (time 1772167613.66 total 63.94 last 333.55) - open/close 27107 (time 1772167623.66 total 73.94 last 377.05) - open/close 30000 (time 1772167632.28 total 82.57 last 335.55) - open/close 33395 (time 1772167642.28 total 92.57 last 339.42) - open/close 36464 (time 1772167652.28 total 102.57 last 306.88) - open/close 39688 (time 1772167662.29 total 112.57 last 322.30) - open/close 42632 (time 1772167672.29 total 122.57 last 294.34) - open/close 46097 (time 1772167682.29 total 132.58 last 346.45) - open/close 49836 (time 1772167692.29 total 142.58 last 373.81) - open/close 53787 (time 1772167702.29 total 152.58 last 395.09) - open/close 57689 (time 1772167712.29 total 162.58 last 390.12) - open/close 60000 (time 1772167718.26 total 168.55 last 387.30) - open/close 63624 (time 1772167728.26 total 178.55 last 362.32) total: 64768 open/close in 181.77 seconds: 356.33 ops/second - /unlink 6618 (time 1772167743.19 total 10.00 last 661.78) - /unlink 10000 (time 1772167747.84 total 14.65 last 726.65) - /unlink 15915 (time 1772167757.84 total 24.66 last 591.39) - /unlink 20000 (time 1772167764.68 total 31.49 last 597.83) - /unlink 27034 (time 1772167774.68 total 41.49 last 703.38) - /unlink 30000 (time 1772167780.00 total 46.81 last 557.26) - /unlink 36610 (time 1772167790.00 total 56.81 last 660.91) - /unlink 40000 (time 1772167795.77 total 62.59 last 587.27) - /unlink 44971 (time 1772167805.77 total 72.59 last 497.05) - /unlink 49485 (time 1772167815.78 total 82.59 last 451.30) - /unlink 53836 (time 1772167825.78 total 92.59 last 435.05) - /unlink 58416 (time 1772167835.78 total 102.59 last 457.98) - /unlink 60000 (time 1772167839.45 total 106.26 last 431.26) - /unlink 64663 (time 1772167849.45 total 116.26 last 466.26) total: 64768 /unlink in 116.58 seconds: 555.57 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 - open/close 2508 (time 1772167868.53 total 10.00 last 250.77) - open/close 5169 (time 1772167878.53 total 20.00 last 266.07) - open/close 7739 (time 1772167888.54 total 30.00 last 256.93) - open/close 10000 (time 1772167896.66 total 38.13 last 278.35) - open/close 12397 (time 1772167906.66 total 48.13 last 239.65) - open/close 14890 (time 1772167916.67 total 58.13 last 249.20) - open/close 17502 (time 1772167926.67 total 68.14 last 261.13) - open/close 19947 (time 1772167936.67 total 78.14 last 244.47) - open/close 22989 (time 1772167946.67 total 88.14 last 304.19) - open/close 25462 (time 1772167956.67 total 98.14 last 247.22) - open/close 27763 (time 1772167966.68 total 108.15 last 230.00) - open/close 30000 (time 1772167974.76 total 116.23 last 276.61) - open/close 32635 (time 1772167984.77 total 126.23 last 263.45) - open/close 35144 (time 1772167994.77 total 136.24 last 250.84) - open/close 37608 (time 1772168004.77 total 146.24 last 246.35) - open/close 40000 (time 1772168014.29 total 155.76 last 251.31) - open/close 42141 (time 1772168024.29 total 165.76 last 214.10) - open/close 44383 (time 1772168034.29 total 175.76 last 224.15) - open/close 46914 (time 1772168044.29 total 185.76 last 253.01) - open/close 49220 (time 1772168054.30 total 195.77 last 230.49) - open/close 50000 (time 1772168057.88 total 199.35 last 217.56) - open/close 52161 (time 1772168067.89 total 209.35 last 216.08) - open/close 54457 (time 1772168077.89 total 219.36 last 229.58) - open/close 57154 (time 1772168087.89 total 229.36 last 269.64) - open/close 60000 (time 1772168097.70 total 239.17 last 290.03) - open/close 63122 (time 1772168107.70 total 249.17 last 312.15) total: 64768 open/close in 253.57 seconds: 255.43 ops/second - /unlink 5353 (time 1772168123.70 total 10.00 last 535.28) - /unlink 10000 (time 1772168131.57 total 17.86 last 591.10) - /unlink 15189 (time 1772168141.57 total 27.86 last 518.83) - /unlink 20000 (time 1772168151.02 total 37.32 last 508.76) - /unlink 25060 (time 1772168161.02 total 47.32 last 505.90) - /unlink 30000 (time 1772168170.51 total 56.80 last 520.96) - /unlink 36154 (time 1772168180.51 total 66.80 last 615.36) - /unlink 40000 (time 1772168186.76 total 73.05 last 615.38) - /unlink 46021 (time 1772168196.76 total 83.06 last 601.99) - /unlink 50000 (time 1772168205.54 total 91.84 last 453.07) - /unlink 55145 (time 1772168215.54 total 101.84 last 514.45) - /unlink 60000 (time 1772168224.46 total 110.76 last 544.43) - /unlink 64445 (time 1772168234.46 total 120.76 last 444.46) total: 64768 /unlink in 121.63 seconds: 532.52 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 - open/close 2619 (time 1772168257.31 total 10.00 last 261.84) - open/close 5216 (time 1772168267.31 total 20.00 last 259.66) - open/close 7711 (time 1772168277.31 total 30.01 last 249.42) - open/close 10000 (time 1772168285.72 total 38.42 last 272.18) - open/close 12391 (time 1772168295.72 total 48.42 last 239.09) - open/close 14886 (time 1772168305.72 total 58.42 last 249.48) - open/close 17314 (time 1772168315.73 total 68.42 last 242.73) - open/close 19691 (time 1772168325.73 total 78.42 last 237.61) - open/close 21940 (time 1772168335.73 total 88.43 last 224.87) - open/close 24319 (time 1772168345.73 total 98.43 last 237.89) - open/close 26639 (time 1772168355.73 total 108.43 last 231.97) - open/close 28952 (time 1772168365.73 total 118.43 last 231.25) - open/close 30000 (time 1772168370.20 total 122.90 last 234.69) - open/close 32524 (time 1772168380.20 total 132.90 last 252.36) - open/close 34943 (time 1772168390.21 total 142.90 last 241.79) - open/close 37264 (time 1772168400.21 total 152.91 last 232.00) - open/close 39586 (time 1772168410.21 total 162.91 last 232.19) - open/close 42314 (time 1772168420.21 total 172.91 last 272.76) - open/close 44908 (time 1772168430.22 total 182.91 last 259.30) - open/close 47274 (time 1772168440.22 total 192.92 last 236.51) - open/close 49871 (time 1772168450.22 total 202.92 last 259.62) - open/close 52586 (time 1772168460.22 total 212.92 last 271.47) - open/close 55604 (time 1772168470.22 total 222.92 last 301.75) - open/close 59395 (time 1772168480.23 total 232.92 last 379.07) - open/close 62810 (time 1772168490.23 total 242.92 last 341.46) total: 64768 open/close in 250.35 seconds: 258.71 ops/second - /unlink 5237 (time 1772168509.67 total 10.00 last 523.63) - /unlink 10000 (time 1772168519.13 total 19.46 last 503.72) - /unlink 15233 (time 1772168529.13 total 29.46 last 523.21) - /unlink 20000 (time 1772168536.44 total 36.77 last 651.91) - /unlink 26503 (time 1772168546.44 total 46.77 last 650.24) - /unlink 30000 (time 1772168553.14 total 53.47 last 522.14) - /unlink 36278 (time 1772168563.14 total 63.47 last 627.76) - /unlink 40000 (time 1772168569.79 total 70.12 last 559.83) - /unlink 45861 (time 1772168579.79 total 80.12 last 586.05) - /unlink 50000 (time 1772168587.98 total 88.31 last 505.49) - /unlink 54681 (time 1772168597.98 total 98.31 last 468.06) - /unlink 58824 (time 1772168607.98 total 108.31 last 414.23) - /unlink 60000 (time 1772168610.83 total 111.16 last 411.91) total: 64768 /unlink in 120.72 seconds: 536.52 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 - open/close 2899 (time 1772168637.53 total 10.00 last 289.84) - open/close 5930 (time 1772168647.53 total 20.00 last 303.06) - open/close 8332 (time 1772168657.53 total 30.01 last 240.15) - open/close 10000 (time 1772168664.42 total 36.89 last 242.34) - open/close 12311 (time 1772168674.42 total 46.89 last 231.06) - open/close 14813 (time 1772168684.42 total 56.89 last 250.13) - open/close 17200 (time 1772168694.42 total 66.89 last 238.67) - open/close 20000 (time 1772168704.09 total 76.56 last 289.70) - open/close 22937 (time 1772168714.09 total 86.56 last 293.61) - open/close 25433 (time 1772168724.09 total 96.57 last 249.52) - open/close 27773 (time 1772168734.10 total 106.57 last 233.93) - open/close 30000 (time 1772168742.40 total 114.87 last 268.32) - open/close 32524 (time 1772168752.40 total 124.87 last 252.39) - open/close 35090 (time 1772168762.40 total 134.87 last 256.54) - open/close 37602 (time 1772168772.40 total 144.87 last 251.19) - open/close 40000 (time 1772168781.63 total 154.10 last 259.90) - open/close 42663 (time 1772168791.63 total 164.10 last 266.21) - open/close 45027 (time 1772168801.63 total 174.11 last 236.32) - open/close 48141 (time 1772168811.64 total 184.11 last 311.37) - open/close 50000 (time 1772168818.71 total 191.18 last 262.65) - open/close 52114 (time 1772168828.72 total 201.19 last 211.35) - open/close 54036 (time 1772168838.73 total 211.20 last 192.00) - open/close 54936 (time 1772168848.73 total 221.20 last 90.00) - open/close 56094 (time 1772168858.73 total 231.20 last 115.74) - open/close 58733 (time 1772168868.73 total 241.20 last 263.83) - open/close 60000 (time 1772168873.32 total 245.79 last 276.39) - open/close 63832 (time 1772168883.32 total 255.79 last 383.12) total: 64768 open/close in 258.50 seconds: 250.55 ops/second - /unlink 3925 (time 1772168897.72 total 10.03 last 391.27) - /unlink 7322 (time 1772168907.72 total 20.03 last 339.70) - /unlink 10000 (time 1772168913.14 total 25.45 last 493.82) - /unlink 15675 (time 1772168923.14 total 35.45 last 567.49) - /unlink 20000 (time 1772168930.27 total 42.58 last 607.09) - /unlink 27398 (time 1772168940.27 total 52.58 last 739.74) - /unlink 30000 (time 1772168945.47 total 57.78 last 500.23) - /unlink 33980 (time 1772168955.47 total 67.79 last 397.81) - /unlink 37104 (time 1772168965.47 total 77.79 last 312.40) - /unlink 40000 (time 1772168971.92 total 84.24 last 449.01) - /unlink 44550 (time 1772168981.93 total 94.24 last 454.89) - /unlink 47408 (time 1772168991.93 total 104.24 last 285.76) - /unlink 50000 (time 1772168999.41 total 111.72 last 346.53) - /unlink 55179 (time 1772169009.41 total 121.72 last 517.72) - /unlink 60000 (time 1772169016.33 total 128.65 last 696.30) total: 64768 /unlink in 135.11 seconds: 479.37 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 - open/close 5121 (time 1772169043.32 total 10.00 last 512.00) - open/close 9865 (time 1772169053.32 total 20.00 last 474.32) - open/close 15359 (time 1772169063.32 total 30.01 last 549.32) - open/close 20000 (time 1772169071.79 total 38.48 last 547.70) - open/close 25405 (time 1772169081.79 total 48.48 last 540.50) - open/close 30000 (time 1772169089.99 total 56.67 last 560.85) - open/close 35572 (time 1772169099.99 total 66.67 last 557.20) - open/close 40000 (time 1772169108.79 total 75.48 last 502.93) - open/close 45358 (time 1772169118.79 total 85.48 last 535.73) - open/close 50000 (time 1772169127.98 total 94.67 last 505.21) - open/close 54698 (time 1772169137.98 total 104.67 last 469.71) - open/close 59361 (time 1772169147.98 total 114.67 last 466.22) - open/close 63881 (time 1772169157.99 total 124.67 last 451.92) total: 64768 open/close in 126.54 seconds: 511.84 ops/second - /unlink 7486 (time 1772169171.33 total 10.00 last 748.51) - /unlink 10000 (time 1772169174.97 total 13.64 last 691.21) - /unlink 17631 (time 1772169184.97 total 23.64 last 763.08) - /unlink 20000 (time 1772169187.96 total 26.63 last 791.56) - /unlink 27545 (time 1772169197.96 total 36.63 last 754.47) - /unlink 30000 (time 1772169200.85 total 39.52 last 849.43) - /unlink 37607 (time 1772169210.85 total 49.52 last 760.66) - /unlink 40000 (time 1772169213.70 total 52.38 last 838.36) - /unlink 47458 (time 1772169223.71 total 62.38 last 745.72) - /unlink 50000 (time 1772169227.74 total 66.41 last 629.78) - /unlink 56594 (time 1772169237.74 total 76.42 last 659.33) - /unlink 60000 (time 1772169242.70 total 81.37 last 686.90) total: 64768 /unlink in 87.46 seconds: 740.52 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2