== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:42:33 (1773232953) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg146-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2392 (time 1773233012.90 total 10.00 last 239.13) - open/close 5367 (time 1773233022.90 total 20.01 last 297.42) - open/close 7965 (time 1773233032.90 total 30.01 last 259.77) - open/close 10000 (time 1773233042.50 total 39.61 last 211.99) - open/close 11213 (time 1773233052.50 total 49.61 last 121.28) - open/close 12694 (time 1773233062.51 total 59.61 last 148.06) - open/close 14535 (time 1773233072.51 total 69.61 last 184.06) - open/close 17413 (time 1773233082.51 total 79.61 last 287.79) - open/close 20000 (time 1773233090.44 total 87.54 last 326.34) - open/close 23723 (time 1773233100.44 total 97.54 last 372.24) - open/close 27238 (time 1773233110.44 total 107.54 last 351.49) - open/close 30000 (time 1773233119.37 total 116.47 last 309.24) - open/close 33067 (time 1773233129.37 total 126.47 last 306.68) - open/close 36522 (time 1773233139.37 total 136.47 last 345.49) - open/close 39448 (time 1773233149.37 total 146.48 last 292.56) - open/close 42905 (time 1773233159.38 total 156.48 last 345.43) - open/close 45351 (time 1773233169.38 total 166.48 last 244.59) - open/close 47955 (time 1773233179.39 total 176.49 last 260.21) - open/close 50000 (time 1773233189.16 total 186.27 last 209.20) - open/close 52427 (time 1773233199.17 total 196.27 last 242.62) - open/close 55255 (time 1773233209.17 total 206.27 last 282.78) - open/close 57786 (time 1773233219.17 total 216.27 last 253.09) - open/close 60000 (time 1773233227.17 total 224.28 last 276.51) - open/close 64257 (time 1773233237.17 total 234.28 last 425.69) total: 64768 open/close in 235.72 seconds: 274.77 ops/second - /unlink 6342 (time 1773233250.11 total 10.00 last 634.18) - /unlink 10000 (time 1773233257.51 total 17.39 last 494.81) - /unlink 15738 (time 1773233267.51 total 27.39 last 573.77) - /unlink 20000 (time 1773233274.89 total 34.78 last 577.07) - /unlink 24988 (time 1773233284.89 total 44.78 last 498.75) - /unlink 29797 (time 1773233294.89 total 54.78 last 480.83) - /unlink 34517 (time 1773233304.90 total 64.78 last 471.95) - /unlink 38070 (time 1773233314.90 total 74.79 last 355.18) - /unlink 40000 (time 1773233318.52 total 78.41 last 532.31) - /unlink 45360 (time 1773233328.52 total 88.41 last 535.98) - /unlink 50000 (time 1773233337.41 total 97.30 last 522.24) - /unlink 54675 (time 1773233347.41 total 107.30 last 467.39) - /unlink 60000 (time 1773233355.02 total 114.91 last 699.92) total: 64768 /unlink in 122.36 seconds: 529.31 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4614 (time 1773233382.83 total 10.00 last 461.33) - open/close 9152 (time 1773233392.84 total 20.00 last 453.72) - open/close 13714 (time 1773233402.84 total 30.00 last 456.15) - open/close 16807 (time 1773233412.84 total 40.01 last 309.25) - open/close 20000 (time 1773233421.48 total 48.65 last 369.42) - open/close 23365 (time 1773233431.49 total 58.65 last 336.38) - open/close 27502 (time 1773233441.49 total 68.65 last 413.66) - open/close 30000 (time 1773233446.76 total 73.92 last 474.14) - open/close 33831 (time 1773233456.77 total 83.94 last 382.61) - open/close 37725 (time 1773233466.77 total 93.94 last 389.39) - open/close 40000 (time 1773233473.31 total 100.48 last 347.77) - open/close 43602 (time 1773233483.31 total 110.48 last 360.16) - open/close 47184 (time 1773233493.31 total 120.48 last 358.14) - open/close 50000 (time 1773233500.98 total 128.15 last 367.07) - open/close 53788 (time 1773233510.99 total 138.15 last 378.70) - open/close 57774 (time 1773233520.99 total 148.15 last 398.56) - open/close 60000 (time 1773233525.69 total 152.86 last 473.19) - open/close 64148 (time 1773233535.69 total 162.86 last 414.72) total: 64768 open/close in 164.46 seconds: 393.82 ops/second - /unlink 8364 (time 1773233548.86 total 10.00 last 836.39) - /unlink 10000 (time 1773233550.88 total 12.02 last 808.25) - /unlink 18092 (time 1773233560.88 total 22.02 last 809.20) - /unlink 20000 (time 1773233563.78 total 24.92 last 658.28) - /unlink 26651 (time 1773233573.78 total 34.92 last 665.07) - /unlink 30000 (time 1773233577.98 total 39.12 last 797.91) - /unlink 37024 (time 1773233587.98 total 49.12 last 702.35) - /unlink 40000 (time 1773233591.96 total 53.10 last 747.30) - /unlink 46247 (time 1773233601.96 total 63.10 last 624.61) - /unlink 50000 (time 1773233607.87 total 69.02 last 634.85) - /unlink 57267 (time 1773233617.87 total 79.02 last 726.65) - /unlink 60000 (time 1773233621.77 total 82.91 last 702.11) total: 64768 /unlink in 89.34 seconds: 724.96 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4852 (time 1773233649.92 total 10.00 last 485.16) - open/close 9252 (time 1773233659.92 total 20.00 last 439.96) - open/close 10000 (time 1773233661.96 total 22.04 last 367.74) - open/close 14341 (time 1773233671.96 total 32.04 last 434.07) - open/close 18404 (time 1773233681.96 total 42.04 last 406.26) - open/close 20000 (time 1773233685.93 total 46.01 last 401.53) - open/close 24629 (time 1773233695.93 total 56.01 last 462.90) - open/close 29574 (time 1773233705.93 total 66.01 last 494.50) - open/close 33936 (time 1773233715.94 total 76.01 last 436.08) - open/close 37803 (time 1773233725.94 total 86.02 last 386.64) - open/close 40000 (time 1773233730.51 total 90.59 last 480.28) - open/close 44565 (time 1773233740.51 total 100.59 last 456.47) - open/close 49110 (time 1773233750.51 total 110.59 last 454.42) - open/close 53762 (time 1773233760.52 total 120.59 last 465.14) - open/close 57896 (time 1773233770.52 total 130.60 last 413.28) - open/close 60000 (time 1773233776.50 total 136.58 last 351.61) - open/close 63744 (time 1773233786.51 total 146.58 last 374.28) total: 64768 open/close in 149.60 seconds: 432.95 ops/second - /unlink 5206 (time 1773233800.99 total 10.00 last 520.54) - /unlink 10000 (time 1773233810.05 total 19.06 last 529.43) - /unlink 16407 (time 1773233820.05 total 29.06 last 640.63) - /unlink 20000 (time 1773233824.61 total 33.62 last 786.70) - /unlink 28094 (time 1773233834.62 total 43.63 last 809.35) - /unlink 30000 (time 1773233836.85 total 45.86 last 851.60) - /unlink 38010 (time 1773233846.85 total 55.86 last 800.99) - /unlink 40000 (time 1773233849.42 total 58.43 last 774.65) - /unlink 48182 (time 1773233859.42 total 68.43 last 818.13) - /unlink 50000 (time 1773233861.74 total 70.75 last 786.33) - /unlink 57797 (time 1773233871.74 total 80.75 last 779.69) - /unlink 60000 (time 1773233875.01 total 84.02 last 673.11) total: 64768 /unlink in 90.11 seconds: 718.73 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4548 (time 1773233897.91 total 10.00 last 454.76) - open/close 8921 (time 1773233907.91 total 20.00 last 437.26) - open/close 10000 (time 1773233910.16 total 22.25 last 479.11) - open/close 14666 (time 1773233920.16 total 32.26 last 466.54) - open/close 18765 (time 1773233930.16 total 42.26 last 409.90) - open/close 20000 (time 1773233932.90 total 44.99 last 452.00) - open/close 24424 (time 1773233942.90 total 54.99 last 442.39) - open/close 28664 (time 1773233952.90 total 64.99 last 423.97) - open/close 30000 (time 1773233956.11 total 68.20 last 416.05) - open/close 34286 (time 1773233966.11 total 78.20 last 428.56) - open/close 38414 (time 1773233976.11 total 88.20 last 412.71) - open/close 40000 (time 1773233980.23 total 92.32 last 385.36) - open/close 43845 (time 1773233990.23 total 102.32 last 384.41) - open/close 47688 (time 1773234000.23 total 112.32 last 384.26) - open/close 50000 (time 1773234005.80 total 117.89 last 415.04) - open/close 54477 (time 1773234015.80 total 127.89 last 447.69) - open/close 58405 (time 1773234025.80 total 137.90 last 392.68) - open/close 60000 (time 1773234029.78 total 141.88 last 400.79) - open/close 64586 (time 1773234039.78 total 151.88 last 458.59) total: 64768 open/close in 152.28 seconds: 425.32 ops/second - /unlink 7995 (time 1773234051.52 total 10.00 last 799.41) - /unlink 10000 (time 1773234054.17 total 12.65 last 757.86) - /unlink 19026 (time 1773234064.17 total 22.65 last 902.53) - /unlink 27002 (time 1773234074.17 total 32.65 last 797.59) - /unlink 30000 (time 1773234078.09 total 36.57 last 763.50) - /unlink 37793 (time 1773234088.09 total 46.57 last 779.29) - /unlink 40000 (time 1773234090.98 total 49.46 last 764.04) - /unlink 47372 (time 1773234100.98 total 59.46 last 737.19) - /unlink 50000 (time 1773234104.21 total 62.69 last 814.89) - /unlink 57846 (time 1773234114.21 total 72.69 last 784.53) - /unlink 60000 (time 1773234116.99 total 75.47 last 773.72) total: 64768 /unlink in 81.51 seconds: 794.56 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4528 (time 1773234142.66 total 10.00 last 452.77) - open/close 9088 (time 1773234152.66 total 20.00 last 455.97) - open/close 13610 (time 1773234162.66 total 30.00 last 452.19) - open/close 18060 (time 1773234172.66 total 40.00 last 444.90) - open/close 20000 (time 1773234176.92 total 44.26 last 456.20) - open/close 24455 (time 1773234186.92 total 54.26 last 445.41) - open/close 29063 (time 1773234196.92 total 64.26 last 460.71) - open/close 30000 (time 1773234199.06 total 66.40 last 438.40) - open/close 34286 (time 1773234209.06 total 76.40 last 428.59) - open/close 39039 (time 1773234219.06 total 86.40 last 475.26) - open/close 43780 (time 1773234229.06 total 96.40 last 474.10) - open/close 48427 (time 1773234239.06 total 106.40 last 464.69) - open/close 50000 (time 1773234242.52 total 109.86 last 454.07) - open/close 54680 (time 1773234252.52 total 119.86 last 467.98) - open/close 59338 (time 1773234262.54 total 129.88 last 464.94) - open/close 63985 (time 1773234272.54 total 139.88 last 464.64) total: 64768 open/close in 141.55 seconds: 457.56 ops/second - /unlink 7865 (time 1773234285.63 total 10.00 last 786.49) - /unlink 10000 (time 1773234288.34 total 12.71 last 788.39) - /unlink 17873 (time 1773234298.34 total 22.71 last 787.30) - /unlink 20000 (time 1773234301.11 total 25.47 last 769.13) - /unlink 27980 (time 1773234311.11 total 35.47 last 797.95) - /unlink 30000 (time 1773234313.68 total 38.04 last 786.17) - /unlink 37924 (time 1773234323.68 total 48.04 last 792.40) - /unlink 40000 (time 1773234326.39 total 50.76 last 764.78) - /unlink 48090 (time 1773234336.39 total 60.76 last 808.94) - /unlink 50000 (time 1773234338.72 total 63.09 last 818.97) - /unlink 58050 (time 1773234348.72 total 73.09 last 804.94) - /unlink 60000 (time 1773234351.13 total 75.50 last 809.11) total: 64768 /unlink in 81.36 seconds: 796.08 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2