== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:03:39 (1768140219) Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f) oleg320-server: oleg320-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg320-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg320-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1901 (time 1768140281.64 total 10.00 last 190.06) - open/close 3664 (time 1768140291.65 total 20.01 last 176.23) - open/close 5179 (time 1768140301.65 total 30.01 last 151.41) - open/close 6861 (time 1768140311.65 total 40.01 last 168.19) - open/close 8970 (time 1768140321.66 total 50.01 last 210.85) - open/close 10000 (time 1768140327.06 total 55.41 last 190.80) - open/close 12231 (time 1768140337.06 total 65.41 last 223.07) - open/close 14701 (time 1768140347.06 total 75.41 last 246.99) - open/close 17498 (time 1768140357.06 total 85.42 last 279.64) - open/close 19625 (time 1768140367.06 total 95.42 last 212.67) - open/close 23258 (time 1768140377.06 total 105.42 last 363.18) - open/close 25737 (time 1768140387.07 total 115.42 last 247.86) - open/close 27971 (time 1768140397.07 total 125.43 last 223.35) - open/close 30000 (time 1768140405.45 total 133.81 last 242.08) - open/close 32165 (time 1768140415.45 total 143.81 last 216.44) - open/close 35432 (time 1768140425.45 total 153.81 last 326.65) - open/close 39168 (time 1768140435.45 total 163.81 last 373.58) - open/close 40000 (time 1768140437.89 total 166.25 last 341.85) - open/close 43354 (time 1768140447.89 total 176.25 last 335.32) - open/close 46987 (time 1768140457.89 total 186.25 last 363.26) - open/close 50000 (time 1768140466.81 total 195.17 last 337.82) - open/close 53299 (time 1768140476.81 total 205.17 last 329.85) - open/close 56130 (time 1768140486.81 total 215.17 last 283.08) - open/close 58509 (time 1768140496.81 total 225.17 last 237.89) - open/close 60000 (time 1768140502.18 total 230.54 last 277.70) - open/close 62453 (time 1768140512.19 total 240.54 last 245.21) total: 64768 open/close in 247.85 seconds: 261.32 ops/second - /unlink 4293 (time 1768140531.05 total 10.00 last 429.29) - /unlink 9001 (time 1768140541.05 total 20.00 last 470.73) - /unlink 14976 (time 1768140551.05 total 30.00 last 597.49) - /unlink 20000 (time 1768140560.77 total 39.72 last 516.93) - /unlink 25611 (time 1768140570.77 total 49.72 last 560.98) - /unlink 29736 (time 1768140580.77 total 59.72 last 412.47) - /unlink 34348 (time 1768140590.78 total 69.73 last 461.13) - /unlink 40000 (time 1768140600.64 total 79.59 last 572.94) - /unlink 45669 (time 1768140610.64 total 89.59 last 566.83) - /unlink 50000 (time 1768140618.20 total 97.15 last 572.76) - /unlink 55456 (time 1768140628.20 total 107.15 last 545.59) - /unlink 60000 (time 1768140635.96 total 114.91 last 586.11) total: 64768 /unlink in 122.92 seconds: 526.93 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 3098 (time 1768140663.57 total 10.00 last 309.71) - open/close 6418 (time 1768140673.57 total 20.00 last 331.94) - open/close 9757 (time 1768140683.57 total 30.01 last 333.85) - open/close 13773 (time 1768140693.57 total 40.01 last 401.51) - open/close 17558 (time 1768140703.57 total 50.01 last 378.48) - open/close 20000 (time 1768140710.25 total 56.69 last 365.55) - open/close 23982 (time 1768140720.26 total 66.69 last 398.14) - open/close 28772 (time 1768140730.26 total 76.69 last 478.99) - open/close 30000 (time 1768140733.18 total 79.61 last 420.19) - open/close 34258 (time 1768140743.18 total 89.61 last 425.76) - open/close 38566 (time 1768140753.18 total 99.62 last 430.71) - open/close 40000 (time 1768140756.81 total 103.25 last 394.61) - open/close 44442 (time 1768140766.82 total 113.25 last 444.16) - open/close 49617 (time 1768140776.82 total 123.25 last 517.44) - open/close 54776 (time 1768140786.82 total 133.25 last 515.90) - open/close 59938 (time 1768140796.82 total 143.25 last 516.19) total: 64768 open/close in 152.50 seconds: 424.71 ops/second - /unlink 8401 (time 1768140817.37 total 10.00 last 840.09) - /unlink 17000 (time 1768140827.37 total 20.00 last 859.88) - /unlink 20000 (time 1768140831.17 total 23.80 last 789.50) - /unlink 28560 (time 1768140841.17 total 33.80 last 855.96) - /unlink 37285 (time 1768140851.17 total 43.80 last 872.42) - /unlink 40000 (time 1768140854.11 total 46.74 last 922.81) - /unlink 49426 (time 1768140864.11 total 56.74 last 942.53) - /unlink 57987 (time 1768140874.11 total 66.75 last 856.04) - /unlink 60000 (time 1768140876.64 total 69.28 last 795.49) total: 64768 /unlink in 75.00 seconds: 863.55 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 4792 (time 1768140900.69 total 10.00 last 479.16) - open/close 9561 (time 1768140910.69 total 20.00 last 476.89) - open/close 14561 (time 1768140920.69 total 30.00 last 499.97) - open/close 19631 (time 1768140930.70 total 40.00 last 506.94) - open/close 24871 (time 1768140940.70 total 50.00 last 523.92) - open/close 30000 (time 1768140950.47 total 59.77 last 524.95) - open/close 34884 (time 1768140960.47 total 69.77 last 488.40) - open/close 40000 (time 1768140969.88 total 79.19 last 543.26) - open/close 45041 (time 1768140979.89 total 89.19 last 504.01) - open/close 49824 (time 1768140989.89 total 99.19 last 478.29) - open/close 54756 (time 1768140999.89 total 109.20 last 493.10) - open/close 59510 (time 1768141009.89 total 119.20 last 475.29) - open/close 63901 (time 1768141019.89 total 129.20 last 438.91) total: 64768 open/close in 131.43 seconds: 492.80 ops/second - /unlink 8005 (time 1768141033.61 total 10.00 last 800.40) - /unlink 10000 (time 1768141036.17 total 12.56 last 779.29) - /unlink 18010 (time 1768141046.17 total 22.56 last 800.93) - /unlink 20000 (time 1768141048.55 total 24.94 last 836.93) - /unlink 27470 (time 1768141058.55 total 34.94 last 746.95) - /unlink 30000 (time 1768141062.03 total 38.42 last 727.04) - /unlink 37929 (time 1768141072.03 total 48.42 last 792.84) - /unlink 40000 (time 1768141074.69 total 51.08 last 778.53) - /unlink 47798 (time 1768141084.69 total 61.08 last 779.74) - /unlink 50000 (time 1768141087.37 total 63.76 last 821.67) - /unlink 57943 (time 1768141097.37 total 73.76 last 794.22) - /unlink 60000 (time 1768141099.97 total 76.35 last 793.86) total: 64768 /unlink in 82.31 seconds: 786.91 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 4980 (time 1768141124.89 total 10.00 last 497.96) - open/close 9992 (time 1768141134.89 total 20.00 last 501.18) - open/close 15361 (time 1768141144.89 total 30.00 last 536.85) - open/close 20000 (time 1768141153.54 total 38.65 last 536.16) - open/close 25374 (time 1768141163.54 total 48.66 last 537.35) - open/close 30000 (time 1768141172.97 total 58.08 last 490.78) - open/close 34800 (time 1768141182.97 total 68.08 last 479.97) - open/close 39596 (time 1768141192.97 total 78.08 last 479.56) - open/close 44425 (time 1768141202.97 total 88.08 last 482.86) - open/close 49275 (time 1768141212.97 total 98.08 last 484.96) - open/close 54218 (time 1768141222.97 total 108.08 last 494.28) - open/close 59081 (time 1768141232.97 total 118.09 last 486.21) - open/close 63908 (time 1768141242.97 total 128.09 last 482.70) total: 64768 open/close in 129.88 seconds: 498.69 ops/second - /unlink 8003 (time 1768141256.15 total 10.00 last 800.23) - /unlink 10000 (time 1768141258.63 total 12.48 last 806.17) - /unlink 18456 (time 1768141268.63 total 22.48 last 845.54) - /unlink 27148 (time 1768141278.63 total 32.48 last 869.15) - /unlink 30000 (time 1768141282.00 total 35.85 last 846.04) - /unlink 38356 (time 1768141292.00 total 45.85 last 835.54) - /unlink 40000 (time 1768141294.02 total 47.87 last 815.81) - /unlink 47726 (time 1768141304.02 total 57.87 last 772.59) - /unlink 50000 (time 1768141306.90 total 60.75 last 789.32) - /unlink 58347 (time 1768141316.90 total 70.75 last 834.70) - /unlink 60000 (time 1768141319.03 total 72.88 last 775.70) total: 64768 /unlink in 78.70 seconds: 822.99 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 4808 (time 1768141345.72 total 10.00 last 480.72) - open/close 9504 (time 1768141355.72 total 20.00 last 469.56) - open/close 14408 (time 1768141365.72 total 30.00 last 490.34) - open/close 19270 (time 1768141375.73 total 40.01 last 486.10) - open/close 24249 (time 1768141385.73 total 50.01 last 497.90) - open/close 29055 (time 1768141395.73 total 60.01 last 480.58) - open/close 33991 (time 1768141405.73 total 70.01 last 493.54) - open/close 38895 (time 1768141415.73 total 80.01 last 490.35) - open/close 40000 (time 1768141417.92 total 82.20 last 504.74) - open/close 45010 (time 1768141427.92 total 92.20 last 500.99) - open/close 49979 (time 1768141437.92 total 102.20 last 496.85) - open/close 54772 (time 1768141447.92 total 112.20 last 479.24) - open/close 59273 (time 1768141457.92 total 122.20 last 450.09) - open/close 64319 (time 1768141467.92 total 132.20 last 504.56) total: 64768 open/close in 133.10 seconds: 486.61 ops/second - /unlink 8108 (time 1768141480.13 total 10.00 last 810.73) - /unlink 10000 (time 1768141482.57 total 12.44 last 774.69) - /unlink 18248 (time 1768141492.58 total 22.45 last 824.56) - /unlink 20000 (time 1768141494.64 total 24.51 last 847.97) - /unlink 27988 (time 1768141504.64 total 34.51 last 798.78) - /unlink 30000 (time 1768141507.09 total 36.96 last 821.76) - /unlink 38190 (time 1768141517.09 total 46.96 last 818.99) - /unlink 40000 (time 1768141519.52 total 49.39 last 744.43) - /unlink 48104 (time 1768141529.52 total 59.39 last 810.34) - /unlink 50000 (time 1768141532.06 total 61.93 last 746.65) - /unlink 58031 (time 1768141542.06 total 71.93 last 803.03) - /unlink 60000 (time 1768141544.58 total 74.45 last 782.68) total: 64768 /unlink in 81.26 seconds: 797.02 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2