== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:14:01 (1755695641) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3176 (time 1755695702.75 total 10.00 last 317.57) - open/close 6813 (time 1755695712.75 total 20.00 last 363.66) - open/close 9707 (time 1755695722.75 total 30.00 last 289.34) - open/close 12421 (time 1755695732.75 total 40.01 last 271.30) - open/close 14967 (time 1755695742.75 total 50.01 last 254.59) - open/close 17283 (time 1755695752.76 total 60.01 last 231.49) - open/close 19661 (time 1755695762.76 total 70.01 last 237.80) - open/close 22079 (time 1755695772.76 total 80.01 last 241.77) - open/close 24909 (time 1755695782.76 total 90.02 last 282.95) - open/close 27393 (time 1755695792.76 total 100.02 last 248.35) - open/close 29670 (time 1755695802.77 total 110.02 last 227.62) - open/close 32300 (time 1755695812.77 total 120.02 last 263.00) - open/close 34888 (time 1755695822.77 total 130.02 last 258.79) - open/close 37595 (time 1755695832.77 total 140.02 last 270.63) - open/close 40000 (time 1755695841.96 total 149.21 last 261.76) - open/close 42722 (time 1755695851.96 total 159.21 last 272.17) - open/close 45422 (time 1755695861.96 total 169.21 last 269.99) - open/close 48381 (time 1755695871.96 total 179.21 last 295.88) - open/close 50000 (time 1755695877.39 total 184.65 last 298.07) - open/close 52814 (time 1755695887.39 total 194.65 last 281.38) - open/close 55463 (time 1755695897.39 total 204.65 last 264.81) - open/close 58257 (time 1755695907.39 total 214.65 last 279.38) - open/close 60000 (time 1755695913.17 total 220.43 last 301.61) - open/close 63307 (time 1755695923.17 total 230.43 last 330.69) total: 64768 open/close in 235.00 seconds: 275.61 ops/second - /unlink 5771 (time 1755695939.77 total 10.00 last 577.06) - /unlink 10000 (time 1755695947.17 total 17.40 last 571.41) - /unlink 15441 (time 1755695957.17 total 27.40 last 544.07) - /unlink 20000 (time 1755695966.09 total 36.32 last 511.16) - /unlink 24997 (time 1755695976.09 total 46.32 last 499.60) - /unlink 29719 (time 1755695986.09 total 56.32 last 472.15) - /unlink 34914 (time 1755695996.09 total 66.33 last 519.40) - /unlink 39858 (time 1755696006.10 total 76.33 last 494.30) - /unlink 44646 (time 1755696016.10 total 86.33 last 478.75) - /unlink 49645 (time 1755696026.10 total 96.33 last 499.89) - /unlink 54399 (time 1755696036.10 total 106.33 last 475.40) - /unlink 58712 (time 1755696046.10 total 116.33 last 431.27) - /unlink 60000 (time 1755696048.68 total 118.91 last 499.18) total: 64768 /unlink in 127.87 seconds: 506.53 ops/second Waiting for MDT destroys to complete oleg118-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 - open/close 2719 (time 1755696079.92 total 10.00 last 271.84) - open/close 5273 (time 1755696089.93 total 20.00 last 255.35) - open/close 8415 (time 1755696099.93 total 30.00 last 314.18) - open/close 10000 (time 1755696105.42 total 35.50 last 288.61) - open/close 12775 (time 1755696115.42 total 45.50 last 277.40) - open/close 15274 (time 1755696125.42 total 55.50 last 249.88) - open/close 18118 (time 1755696135.42 total 65.50 last 284.39) - open/close 20000 (time 1755696142.50 total 72.58 last 265.98) - open/close 23183 (time 1755696152.50 total 82.58 last 318.20) - open/close 26234 (time 1755696162.50 total 92.58 last 305.09) - open/close 28787 (time 1755696172.50 total 102.58 last 255.27) - open/close 30000 (time 1755696177.45 total 107.53 last 245.31) - open/close 32813 (time 1755696187.45 total 117.53 last 281.29) - open/close 35253 (time 1755696197.45 total 127.53 last 243.94) - open/close 37660 (time 1755696207.45 total 137.53 last 240.66) - open/close 40000 (time 1755696216.63 total 146.71 last 254.84) - open/close 43424 (time 1755696226.63 total 156.71 last 342.38) - open/close 46345 (time 1755696236.64 total 166.71 last 292.07) - open/close 49777 (time 1755696246.64 total 176.72 last 343.19) - open/close 53243 (time 1755696256.64 total 186.72 last 346.56) - open/close 56415 (time 1755696266.64 total 196.72 last 317.17) - open/close 59498 (time 1755696276.64 total 206.72 last 308.24) - open/close 62220 (time 1755696286.64 total 216.72 last 272.09) total: 64768 open/close in 225.85 seconds: 286.77 ops/second - /unlink 6562 (time 1755696307.51 total 10.00 last 656.17) - /unlink 10000 (time 1755696313.75 total 16.23 last 551.64) - /unlink 16314 (time 1755696323.75 total 26.23 last 631.34) - /unlink 20000 (time 1755696330.11 total 32.60 last 579.22) - /unlink 25392 (time 1755696340.11 total 42.60 last 539.19) - /unlink 30000 (time 1755696348.56 total 51.05 last 545.32) - /unlink 34714 (time 1755696358.56 total 61.05 last 471.30) - /unlink 39203 (time 1755696368.56 total 71.05 last 448.85) - /unlink 44248 (time 1755696378.57 total 81.05 last 504.41) - /unlink 50000 (time 1755696386.75 total 89.23 last 703.00) - /unlink 58394 (time 1755696396.75 total 99.24 last 839.36) total: 64768 /unlink in 106.90 seconds: 605.89 ops/second Waiting for MDT destroys to complete oleg118-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 - open/close 3462 (time 1755696445.14 total 10.00 last 346.15) - open/close 7317 (time 1755696455.14 total 20.00 last 385.43) - open/close 10000 (time 1755696463.18 total 28.05 last 333.57) - open/close 13443 (time 1755696473.18 total 38.05 last 344.26) - open/close 16724 (time 1755696483.19 total 48.06 last 327.69) - open/close 19526 (time 1755696493.20 total 58.06 last 280.13) - open/close 23479 (time 1755696503.20 total 68.07 last 395.14) - open/close 27153 (time 1755696513.20 total 78.07 last 367.28) - open/close 30000 (time 1755696521.31 total 86.17 last 351.32) - open/close 33484 (time 1755696531.31 total 96.17 last 348.37) - open/close 37014 (time 1755696541.31 total 106.18 last 352.94) - open/close 39747 (time 1755696551.31 total 116.18 last 273.21) - open/close 42775 (time 1755696561.32 total 126.18 last 302.71) - open/close 46128 (time 1755696571.32 total 136.18 last 335.25) - open/close 49986 (time 1755696581.32 total 146.18 last 385.77) - open/close 53854 (time 1755696591.32 total 156.19 last 386.76) - open/close 57358 (time 1755696601.32 total 166.19 last 350.36) - open/close 60000 (time 1755696607.41 total 172.28 last 433.83) - open/close 63377 (time 1755696617.41 total 182.28 last 337.64) total: 64768 open/close in 186.34 seconds: 347.58 ops/second - /unlink 5473 (time 1755696633.06 total 10.00 last 547.19) - /unlink 10000 (time 1755696639.65 total 16.59 last 687.14) - /unlink 17311 (time 1755696649.65 total 26.59 last 731.09) - /unlink 20000 (time 1755696653.52 total 30.46 last 694.26) - /unlink 26305 (time 1755696663.52 total 40.47 last 630.39) - /unlink 30000 (time 1755696670.35 total 47.30 last 540.78) - /unlink 35890 (time 1755696680.35 total 57.30 last 588.99) - /unlink 40000 (time 1755696688.17 total 65.12 last 525.54) - /unlink 45526 (time 1755696698.17 total 75.12 last 552.58) - /unlink 50000 (time 1755696706.02 total 82.97 last 570.09) - /unlink 56783 (time 1755696716.02 total 92.97 last 678.23) - /unlink 60000 (time 1755696720.09 total 97.03 last 791.85) total: 64768 /unlink in 104.27 seconds: 621.18 ops/second Waiting for MDT destroys to complete oleg118-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 - open/close 2886 (time 1755696758.73 total 10.00 last 288.53) - open/close 6291 (time 1755696768.73 total 20.00 last 340.47) - open/close 9750 (time 1755696778.74 total 30.01 last 345.84) - open/close 13240 (time 1755696788.74 total 40.01 last 348.97) - open/close 16571 (time 1755696798.75 total 50.02 last 332.79) - open/close 18352 (time 1755696808.75 total 60.02 last 178.07) - open/close 20000 (time 1755696817.46 total 68.73 last 189.12) - open/close 24853 (time 1755696827.46 total 78.73 last 485.25) - open/close 29564 (time 1755696837.46 total 88.73 last 471.08) - open/close 34197 (time 1755696847.46 total 98.73 last 463.28) - open/close 38793 (time 1755696857.46 total 108.73 last 459.59) - open/close 40000 (time 1755696860.10 total 111.37 last 458.44) - open/close 44685 (time 1755696870.10 total 121.37 last 468.44) - open/close 49104 (time 1755696880.10 total 131.37 last 441.83) - open/close 50000 (time 1755696882.13 total 133.40 last 440.99) - open/close 54434 (time 1755696892.13 total 143.40 last 443.39) - open/close 58992 (time 1755696902.13 total 153.40 last 455.71) - open/close 60000 (time 1755696904.34 total 155.61 last 457.19) - open/close 64453 (time 1755696914.34 total 165.61 last 445.23) total: 64768 open/close in 166.28 seconds: 389.51 ops/second - /unlink 7700 (time 1755696926.45 total 10.00 last 769.93) - /unlink 10000 (time 1755696929.39 total 12.94 last 782.53) - /unlink 17667 (time 1755696939.39 total 22.94 last 766.66) - /unlink 20000 (time 1755696942.41 total 25.97 last 771.38) - /unlink 27763 (time 1755696952.41 total 35.97 last 776.21) - /unlink 30000 (time 1755696955.11 total 38.66 last 830.40) - /unlink 36821 (time 1755696965.11 total 48.66 last 682.10) - /unlink 40000 (time 1755696968.99 total 52.54 last 819.47) - /unlink 46548 (time 1755696978.99 total 62.54 last 654.71) - /unlink 50000 (time 1755696983.61 total 67.16 last 746.92) - /unlink 55993 (time 1755696993.61 total 77.16 last 599.28) - /unlink 60000 (time 1755696998.58 total 82.14 last 805.61) total: 64768 /unlink in 90.23 seconds: 717.85 ops/second Waiting for MDT destroys to complete oleg118-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 - open/close 3378 (time 1755697065.70 total 10.00 last 337.72) - open/close 6233 (time 1755697075.70 total 20.00 last 285.47) - open/close 9680 (time 1755697085.70 total 30.00 last 344.66) - open/close 13108 (time 1755697095.70 total 40.01 last 342.76) - open/close 16206 (time 1755697105.70 total 50.01 last 309.73) - open/close 19535 (time 1755697115.70 total 60.01 last 332.84) - open/close 22099 (time 1755697125.71 total 70.01 last 256.36) - open/close 23902 (time 1755697135.71 total 80.02 last 180.17) - open/close 25879 (time 1755697145.72 total 90.02 last 197.61) - open/close 28201 (time 1755697155.75 total 100.06 last 231.38) - open/close 30000 (time 1755697160.94 total 105.24 last 347.19) - open/close 34228 (time 1755697170.94 total 115.24 last 422.74) - open/close 39396 (time 1755697180.94 total 125.24 last 516.73) - open/close 43602 (time 1755697190.94 total 135.24 last 420.54) - open/close 46999 (time 1755697200.94 total 145.25 last 339.66) - open/close 50000 (time 1755697207.42 total 151.73 last 463.08) - open/close 54485 (time 1755697217.42 total 161.73 last 448.43) - open/close 59030 (time 1755697227.42 total 171.73 last 454.49) - open/close 60000 (time 1755697229.74 total 174.05 last 418.36) - open/close 64243 (time 1755697239.74 total 184.05 last 424.30) total: 64768 open/close in 185.11 seconds: 349.90 ops/second - /unlink 7042 (time 1755697252.09 total 10.00 last 704.15) - /unlink 10000 (time 1755697255.99 total 13.90 last 759.08) - /unlink 17748 (time 1755697265.99 total 23.90 last 774.60) - /unlink 20000 (time 1755697269.78 total 27.69 last 594.98) - /unlink 25652 (time 1755697279.78 total 37.69 last 565.17) - /unlink 30000 (time 1755697286.91 total 44.82 last 609.81) - /unlink 35689 (time 1755697296.91 total 54.82 last 568.87) - /unlink 40000 (time 1755697305.36 total 63.27 last 510.08) - /unlink 46810 (time 1755697315.36 total 73.27 last 680.90) - /unlink 50000 (time 1755697319.81 total 77.72 last 717.02) - /unlink 58023 (time 1755697329.81 total 87.72 last 802.26) - /unlink 60000 (time 1755697332.40 total 90.31 last 762.47) total: 64768 /unlink in 97.05 seconds: 667.38 ops/second Waiting for MDT destroys to complete oleg118-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2