== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:00:58 (1759813258) Stopping clients: oleg454-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg454-client.virtnet /mnt/lustre2 (opts:-f) oleg454-server: oleg454-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg454-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg454-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 805 (time 1759813352.77 total 10.01 last 80.43) - open/close 1699 (time 1759813362.78 total 20.02 last 89.33) - open/close 2528 (time 1759813372.79 total 30.02 last 82.83) - open/close 3374 (time 1759813382.79 total 40.03 last 84.60) - open/close 4210 (time 1759813392.80 total 50.04 last 83.48) - open/close 5024 (time 1759813402.81 total 60.05 last 81.34) - open/close 5882 (time 1759813412.82 total 70.05 last 85.76) - open/close 6758 (time 1759813422.82 total 80.05 last 87.60) - open/close 7592 (time 1759813432.82 total 90.06 last 83.36) - open/close 8762 (time 1759813442.82 total 100.06 last 116.99) - open/close 9937 (time 1759813452.83 total 110.06 last 117.45) - open/close 11194 (time 1759813462.83 total 120.06 last 125.69) - open/close 12848 (time 1759813472.83 total 130.07 last 165.35) - open/close 14604 (time 1759813482.83 total 140.07 last 175.59) - open/close 15858 (time 1759813492.83 total 150.07 last 125.35) - open/close 17137 (time 1759813502.84 total 160.07 last 127.87) - open/close 18887 (time 1759813512.84 total 170.07 last 174.96) - open/close 20000 (time 1759813518.27 total 175.51 last 204.88) - open/close 21927 (time 1759813528.27 total 185.51 last 192.68) - open/close 23563 (time 1759813538.27 total 195.51 last 163.57) - open/close 25252 (time 1759813548.28 total 205.52 last 168.80) - open/close 27014 (time 1759813558.29 total 215.52 last 176.09) - open/close 28717 (time 1759813568.29 total 225.52 last 170.26) - open/close 30000 (time 1759813577.36 total 234.60 last 141.42) - open/close 31664 (time 1759813587.36 total 244.60 last 166.39) - open/close 33295 (time 1759813597.37 total 254.60 last 163.04) - open/close 35572 (time 1759813607.37 total 264.60 last 227.65) - open/close 38238 (time 1759813617.37 total 274.61 last 266.52) - open/close 40000 (time 1759813622.68 total 279.92 last 331.76) - open/close 42863 (time 1759813632.68 total 289.92 last 286.24) - open/close 45473 (time 1759813642.69 total 299.92 last 260.90) - open/close 48459 (time 1759813652.69 total 309.92 last 298.60) - open/close 50000 (time 1759813657.78 total 315.02 last 302.51) - open/close 53259 (time 1759813667.78 total 325.02 last 325.83) - open/close 56166 (time 1759813677.79 total 335.02 last 290.63) - open/close 59054 (time 1759813687.79 total 345.02 last 288.80) - open/close 60000 (time 1759813690.69 total 347.92 last 325.94) - open/close 63231 (time 1759813700.69 total 357.93 last 323.06) total: 64768 open/close in 362.84 seconds: 178.50 ops/second - /unlink 5397 (time 1759813717.18 total 10.00 last 539.68) - /unlink 9922 (time 1759813727.18 total 20.00 last 452.46) - /unlink 15644 (time 1759813737.18 total 30.00 last 572.20) - /unlink 20000 (time 1759813746.86 total 39.68 last 449.94) - /unlink 25976 (time 1759813756.86 total 49.68 last 597.47) - /unlink 30000 (time 1759813764.77 total 57.60 last 508.59) - /unlink 35105 (time 1759813774.78 total 67.60 last 510.45) - /unlink 40000 (time 1759813783.16 total 75.98 last 584.03) - /unlink 45819 (time 1759813793.16 total 85.98 last 581.66) - /unlink 50000 (time 1759813801.13 total 93.96 last 524.37) - /unlink 55911 (time 1759813811.14 total 103.96 last 591.04) - /unlink 60000 (time 1759813818.64 total 111.46 last 545.21) total: 64768 /unlink in 119.98 seconds: 539.80 ops/second Waiting for MDT destroys to complete oleg454-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 - open/close 2274 (time 1759813846.21 total 10.01 last 227.25) - open/close 4308 (time 1759813856.21 total 20.01 last 203.32) - open/close 7052 (time 1759813866.22 total 30.01 last 274.31) - open/close 10000 (time 1759813873.65 total 37.45 last 396.41) - open/close 13222 (time 1759813883.66 total 47.45 last 322.12) - open/close 16369 (time 1759813893.66 total 57.45 last 314.68) - open/close 19141 (time 1759813903.66 total 67.45 last 277.18) - open/close 20000 (time 1759813906.07 total 69.87 last 355.85) - open/close 23221 (time 1759813916.07 total 79.87 last 322.03) - open/close 26106 (time 1759813926.07 total 89.87 last 288.48) - open/close 29181 (time 1759813936.07 total 99.87 last 307.46) - open/close 32886 (time 1759813946.08 total 109.87 last 370.47) - open/close 36562 (time 1759813956.08 total 119.87 last 367.55) - open/close 40000 (time 1759813965.44 total 129.24 last 367.28) - open/close 43500 (time 1759813975.44 total 139.24 last 349.98) - open/close 47048 (time 1759813985.44 total 149.24 last 354.69) - open/close 50000 (time 1759813993.23 total 157.03 last 378.89) - open/close 54557 (time 1759814003.23 total 167.03 last 455.64) - open/close 58949 (time 1759814013.23 total 177.03 last 439.17) - open/close 60000 (time 1759814015.88 total 179.68 last 397.58) - open/close 64545 (time 1759814025.88 total 189.68 last 454.44) total: 64768 open/close in 190.16 seconds: 340.59 ops/second - /unlink 7650 (time 1759814037.83 total 10.00 last 764.94) - /unlink 10000 (time 1759814041.77 total 13.94 last 596.45) - /unlink 17790 (time 1759814051.77 total 23.94 last 778.99) - /unlink 20000 (time 1759814054.48 total 26.65 last 816.75) - /unlink 28182 (time 1759814064.48 total 36.65 last 818.18) - /unlink 37323 (time 1759814074.48 total 46.65 last 914.06) - /unlink 40000 (time 1759814077.27 total 49.44 last 958.88) - /unlink 49272 (time 1759814087.27 total 59.44 last 927.10) - /unlink 57426 (time 1759814097.27 total 69.44 last 815.30) - /unlink 60000 (time 1759814100.69 total 72.86 last 752.77) total: 64768 /unlink in 78.64 seconds: 823.59 ops/second Waiting for MDT destroys to complete oleg454-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 - open/close 4922 (time 1759814188.03 total 10.00 last 492.17) - open/close 10000 (time 1759814197.85 total 19.82 last 517.37) - open/close 15256 (time 1759814207.85 total 29.82 last 525.55) - open/close 20000 (time 1759814217.41 total 39.38 last 495.94) - open/close 25078 (time 1759814227.42 total 49.38 last 507.71) - open/close 29811 (time 1759814237.42 total 59.38 last 473.28) - open/close 34662 (time 1759814247.42 total 69.39 last 485.00) - open/close 39805 (time 1759814257.42 total 79.39 last 514.23) - open/close 45281 (time 1759814267.42 total 89.39 last 547.50) - open/close 50000 (time 1759814276.89 total 98.86 last 498.24) - open/close 55037 (time 1759814286.89 total 108.86 last 503.65) - open/close 60000 (time 1759814296.41 total 118.38 last 521.52) total: 64768 open/close in 127.49 seconds: 508.02 ops/second - /unlink 8532 (time 1759814316.93 total 10.00 last 853.19) - /unlink 17330 (time 1759814326.93 total 20.00 last 879.75) - /unlink 20000 (time 1759814329.76 total 22.83 last 943.78) - /unlink 27740 (time 1759814339.76 total 32.83 last 773.96) - /unlink 30000 (time 1759814343.28 total 36.35 last 641.86) - /unlink 37471 (time 1759814353.28 total 46.35 last 747.06) - /unlink 40000 (time 1759814356.51 total 49.58 last 784.23) - /unlink 47510 (time 1759814366.51 total 59.58 last 751.00) - /unlink 50000 (time 1759814369.60 total 62.67 last 803.81) - /unlink 58269 (time 1759814379.61 total 72.68 last 826.59) - /unlink 60000 (time 1759814382.19 total 75.26 last 670.00) total: 64768 /unlink in 81.32 seconds: 796.45 ops/second Waiting for MDT destroys to complete oleg454-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 - open/close 4568 (time 1759814440.02 total 10.00 last 456.75) - open/close 9520 (time 1759814450.02 total 20.00 last 495.16) - open/close 13858 (time 1759814460.02 total 30.00 last 433.70) - open/close 17827 (time 1759814470.02 total 40.01 last 396.83) - open/close 20000 (time 1759814476.24 total 46.23 last 349.34) - open/close 23720 (time 1759814486.25 total 56.23 last 371.89) - open/close 26702 (time 1759814496.25 total 66.23 last 298.15) - open/close 30000 (time 1759814503.91 total 73.89 last 430.71) - open/close 34079 (time 1759814513.91 total 83.89 last 407.81) - open/close 38445 (time 1759814523.91 total 93.89 last 436.45) - open/close 40000 (time 1759814527.56 total 97.54 last 426.39) - open/close 45156 (time 1759814537.56 total 107.54 last 515.60) - open/close 49665 (time 1759814547.56 total 117.54 last 450.88) - open/close 54416 (time 1759814557.56 total 127.54 last 475.04) - open/close 59561 (time 1759814567.56 total 137.54 last 514.48) - open/close 64037 (time 1759814577.56 total 147.54 last 447.60) total: 64768 open/close in 149.32 seconds: 433.75 ops/second - /unlink 7186 (time 1759814590.90 total 10.00 last 718.53) - /unlink 10000 (time 1759814594.82 total 13.92 last 718.71) - /unlink 18765 (time 1759814604.82 total 23.92 last 876.49) - /unlink 20000 (time 1759814606.88 total 25.98 last 599.35) - /unlink 26761 (time 1759814616.88 total 35.98 last 676.02) - /unlink 30000 (time 1759814621.62 total 40.72 last 682.74) - /unlink 36872 (time 1759814631.62 total 50.72 last 687.12) - /unlink 40000 (time 1759814636.18 total 55.28 last 686.45) - /unlink 47166 (time 1759814646.18 total 65.28 last 716.54) - /unlink 50000 (time 1759814649.75 total 68.85 last 794.78) - /unlink 57333 (time 1759814659.75 total 78.85 last 733.26) - /unlink 60000 (time 1759814663.13 total 82.23 last 788.62) total: 64768 /unlink in 88.29 seconds: 733.57 ops/second Waiting for MDT destroys to complete oleg454-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 - open/close 4826 (time 1759814717.76 total 10.00 last 482.52) - open/close 9970 (time 1759814727.76 total 20.00 last 514.31) - open/close 15281 (time 1759814737.76 total 30.00 last 531.09) - open/close 19888 (time 1759814747.76 total 40.01 last 460.62) - open/close 24281 (time 1759814757.76 total 50.01 last 439.23) - open/close 28536 (time 1759814767.77 total 60.01 last 425.44) - open/close 30000 (time 1759814771.05 total 63.29 last 445.90) - open/close 34589 (time 1759814781.05 total 73.29 last 458.87) - open/close 39634 (time 1759814791.05 total 83.29 last 504.47) - open/close 44756 (time 1759814801.05 total 93.29 last 512.13) - open/close 49961 (time 1759814811.05 total 103.30 last 520.42) - open/close 54964 (time 1759814821.05 total 113.30 last 500.24) - open/close 59772 (time 1759814831.06 total 123.30 last 480.78) - open/close 64749 (time 1759814841.06 total 133.30 last 497.61) total: 64768 open/close in 133.33 seconds: 485.76 ops/second - /unlink 9105 (time 1759814852.39 total 10.00 last 910.43) - /unlink 18293 (time 1759814862.39 total 20.00 last 918.72) - /unlink 27219 (time 1759814872.40 total 30.00 last 892.52) - /unlink 30000 (time 1759814875.99 total 33.59 last 774.76) - /unlink 38168 (time 1759814885.99 total 43.59 last 816.79) - /unlink 40000 (time 1759814888.25 total 45.85 last 810.43) - /unlink 49132 (time 1759814898.25 total 55.85 last 913.17) - /unlink 58742 (time 1759814908.26 total 65.87 last 959.47) total: 64768 /unlink in 73.14 seconds: 885.47 ops/second Waiting for MDT destroys to complete oleg454-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2