== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:33:58 (1756676038) Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) oleg119-server: oleg119-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg119-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4608 (time 1756676089.94 total 10.00 last 460.70) - open/close 9247 (time 1756676099.94 total 20.00 last 463.87) - open/close 13810 (time 1756676109.95 total 30.00 last 456.25) - open/close 18388 (time 1756676119.95 total 40.00 last 457.78) - open/close 20000 (time 1756676123.57 total 43.62 last 445.39) - open/close 24471 (time 1756676133.57 total 53.62 last 447.03) - open/close 29042 (time 1756676143.57 total 63.63 last 457.07) - open/close 30000 (time 1756676145.94 total 65.99 last 404.69) - open/close 34479 (time 1756676155.94 total 75.99 last 447.87) - open/close 39012 (time 1756676165.94 total 85.99 last 453.27) - open/close 40000 (time 1756676168.10 total 88.16 last 455.94) - open/close 44546 (time 1756676178.10 total 98.16 last 454.53) - open/close 49058 (time 1756676188.11 total 108.16 last 451.13) - open/close 50000 (time 1756676190.20 total 110.26 last 450.55) - open/close 54590 (time 1756676200.21 total 120.27 last 458.51) - open/close 59166 (time 1756676210.21 total 130.27 last 457.51) - open/close 63652 (time 1756676220.21 total 140.27 last 448.58) total: 64768 open/close in 142.73 seconds: 453.79 ops/second - /unlink 7718 (time 1756676234.01 total 10.00 last 771.80) - /unlink 10000 (time 1756676236.97 total 12.96 last 770.76) - /unlink 17754 (time 1756676246.97 total 22.96 last 775.35) - /unlink 20000 (time 1756676249.68 total 25.67 last 830.14) - /unlink 27660 (time 1756676259.68 total 35.67 last 765.90) - /unlink 30000 (time 1756676262.58 total 38.57 last 805.92) - /unlink 37746 (time 1756676272.58 total 48.57 last 774.53) - /unlink 40000 (time 1756676275.45 total 51.44 last 786.13) - /unlink 48526 (time 1756676285.45 total 61.44 last 852.57) - /unlink 57228 (time 1756676295.45 total 71.44 last 870.18) - /unlink 60000 (time 1756676299.00 total 74.99 last 781.21) total: 64768 /unlink in 80.77 seconds: 801.86 ops/second Waiting for MDT destroys to complete oleg119-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 - open/close 4669 (time 1756676356.84 total 10.00 last 466.86) - open/close 9332 (time 1756676366.84 total 20.00 last 466.28) - open/close 13951 (time 1756676376.84 total 30.00 last 461.83) - open/close 18546 (time 1756676386.84 total 40.00 last 459.47) - open/close 20000 (time 1756676389.94 total 43.10 last 469.73) - open/close 24586 (time 1756676399.94 total 53.10 last 458.50) - open/close 29200 (time 1756676409.94 total 63.10 last 461.31) - open/close 33642 (time 1756676419.94 total 73.10 last 444.11) - open/close 38158 (time 1756676429.95 total 83.11 last 451.51) - open/close 40000 (time 1756676434.37 total 87.54 last 415.94) - open/close 44259 (time 1756676444.37 total 97.54 last 425.86) - open/close 48635 (time 1756676454.38 total 107.54 last 437.54) - open/close 50000 (time 1756676457.48 total 110.64 last 440.05) - open/close 54493 (time 1756676467.48 total 120.64 last 449.22) - open/close 59010 (time 1756676477.48 total 130.64 last 451.63) - open/close 60000 (time 1756676479.72 total 132.88 last 441.81) - open/close 64446 (time 1756676489.72 total 142.88 last 444.58) total: 64768 open/close in 143.60 seconds: 451.02 ops/second - /unlink 7883 (time 1756676501.85 total 10.00 last 788.28) - /unlink 10000 (time 1756676504.51 total 12.66 last 796.45) - /unlink 18092 (time 1756676514.51 total 22.66 last 809.20) - /unlink 20000 (time 1756676516.78 total 24.93 last 838.49) - /unlink 28051 (time 1756676526.78 total 34.93 last 805.06) - /unlink 30000 (time 1756676529.21 total 37.36 last 804.19) - /unlink 38144 (time 1756676539.23 total 47.38 last 812.71) - /unlink 40000 (time 1756676541.72 total 49.87 last 744.32) - /unlink 48237 (time 1756676551.72 total 59.87 last 823.66) - /unlink 50000 (time 1756676553.81 total 61.96 last 845.87) - /unlink 57831 (time 1756676563.81 total 71.96 last 783.10) - /unlink 60000 (time 1756676566.64 total 74.79 last 764.90) total: 64768 /unlink in 80.57 seconds: 803.83 ops/second Waiting for MDT destroys to complete oleg119-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 - open/close 4455 (time 1756676649.72 total 10.00 last 445.47) - open/close 9084 (time 1756676659.72 total 20.00 last 462.89) - open/close 13772 (time 1756676669.72 total 30.00 last 468.70) - open/close 18462 (time 1756676679.72 total 40.00 last 468.99) - open/close 20000 (time 1756676683.04 total 43.32 last 463.49) - open/close 24617 (time 1756676693.04 total 53.32 last 461.63) - open/close 29240 (time 1756676703.04 total 63.32 last 462.21) - open/close 33846 (time 1756676713.04 total 73.33 last 460.52) - open/close 38506 (time 1756676723.05 total 83.33 last 465.91) - open/close 40000 (time 1756676726.30 total 86.58 last 459.23) - open/close 44628 (time 1756676736.30 total 96.58 last 462.73) - open/close 49087 (time 1756676746.30 total 106.59 last 445.82) - open/close 50000 (time 1756676748.35 total 108.63 last 446.16) - open/close 54649 (time 1756676758.35 total 118.63 last 464.88) - open/close 59180 (time 1756676768.35 total 128.63 last 453.06) - open/close 60000 (time 1756676770.41 total 130.69 last 398.86) - open/close 63836 (time 1756676780.41 total 140.69 last 383.54) total: 64768 open/close in 143.37 seconds: 451.77 ops/second - /unlink 7317 (time 1756676794.40 total 10.00 last 731.67) - /unlink 10000 (time 1756676797.87 total 13.48 last 771.99) - /unlink 17921 (time 1756676807.87 total 23.48 last 792.07) - /unlink 20000 (time 1756676810.71 total 26.31 last 733.48) - /unlink 27190 (time 1756676820.71 total 36.31 last 718.92) - /unlink 30000 (time 1756676824.16 total 39.76 last 813.93) - /unlink 37933 (time 1756676834.16 total 49.77 last 793.21) - /unlink 40000 (time 1756676836.60 total 52.21 last 847.16) - /unlink 48302 (time 1756676846.60 total 62.21 last 830.20) - /unlink 50000 (time 1756676848.86 total 64.46 last 752.23) - /unlink 58236 (time 1756676858.86 total 74.46 last 823.52) - /unlink 60000 (time 1756676860.96 total 76.57 last 838.41) total: 64768 /unlink in 82.52 seconds: 784.89 ops/second Waiting for MDT destroys to complete oleg119-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 - open/close 4608 (time 1756676919.59 total 10.00 last 460.74) - open/close 9239 (time 1756676929.59 total 20.00 last 463.04) - open/close 13919 (time 1756676939.59 total 30.00 last 467.93) - open/close 18552 (time 1756676949.59 total 40.00 last 463.28) - open/close 20000 (time 1756676952.67 total 43.08 last 470.19) - open/close 24695 (time 1756676962.67 total 53.09 last 469.45) - open/close 29308 (time 1756676972.67 total 63.09 last 461.19) - open/close 33799 (time 1756676982.67 total 73.09 last 449.01) - open/close 38413 (time 1756676992.67 total 83.09 last 461.39) - open/close 40000 (time 1756676996.12 total 86.54 last 459.96) - open/close 44571 (time 1756677006.13 total 96.54 last 457.01) - open/close 49192 (time 1756677016.13 total 106.54 last 462.01) - open/close 53778 (time 1756677026.13 total 116.55 last 458.52) - open/close 58299 (time 1756677036.13 total 126.55 last 452.09) - open/close 60000 (time 1756677040.08 total 130.50 last 430.72) - open/close 64530 (time 1756677050.08 total 140.50 last 452.99) total: 64768 open/close in 141.00 seconds: 459.35 ops/second - /unlink 7928 (time 1756677061.96 total 10.00 last 792.72) - /unlink 10000 (time 1756677064.55 total 12.59 last 800.38) - /unlink 18013 (time 1756677074.55 total 22.59 last 801.22) - /unlink 20000 (time 1756677076.90 total 24.93 last 848.21) - /unlink 27506 (time 1756677086.90 total 34.93 last 750.57) - /unlink 30000 (time 1756677090.27 total 38.30 last 739.78) - /unlink 37172 (time 1756677100.27 total 48.31 last 717.15) - /unlink 40000 (time 1756677104.72 total 52.76 last 634.71) - /unlink 46957 (time 1756677114.72 total 62.76 last 695.65) - /unlink 50000 (time 1756677118.70 total 66.74 last 765.83) - /unlink 57913 (time 1756677128.70 total 76.74 last 791.21) - /unlink 60000 (time 1756677131.61 total 79.65 last 716.06) total: 64768 /unlink in 85.74 seconds: 755.40 ops/second Waiting for MDT destroys to complete oleg119-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 - open/close 4293 (time 1756677207.51 total 10.00 last 429.24) - open/close 8331 (time 1756677217.52 total 20.00 last 403.74) - open/close 10000 (time 1756677222.46 total 24.95 last 337.56) - open/close 13720 (time 1756677232.48 total 34.96 last 371.38) - open/close 18270 (time 1756677242.48 total 44.96 last 455.00) - open/close 20000 (time 1756677246.07 total 48.56 last 481.25) - open/close 24489 (time 1756677256.07 total 58.56 last 448.81) - open/close 28961 (time 1756677266.08 total 68.56 last 447.11) - open/close 30000 (time 1756677268.81 total 71.30 last 380.21) - open/close 34322 (time 1756677278.81 total 81.30 last 432.20) - open/close 38846 (time 1756677288.81 total 91.30 last 452.40) - open/close 40000 (time 1756677291.23 total 93.72 last 476.61) - open/close 44690 (time 1756677301.23 total 103.72 last 468.98) - open/close 49486 (time 1756677311.23 total 113.72 last 479.54) - open/close 54085 (time 1756677321.23 total 123.72 last 459.86) - open/close 58676 (time 1756677331.23 total 133.72 last 459.06) - open/close 60000 (time 1756677334.17 total 136.66 last 450.94) - open/close 64626 (time 1756677344.17 total 146.66 last 462.56) total: 64768 open/close in 146.96 seconds: 440.73 ops/second - /unlink 7907 (time 1756677355.90 total 10.00 last 790.64) - /unlink 10000 (time 1756677358.56 total 12.66 last 786.53) - /unlink 17930 (time 1756677368.56 total 22.66 last 792.96) - /unlink 20000 (time 1756677371.25 total 25.35 last 768.93) - /unlink 27890 (time 1756677381.25 total 35.36 last 788.93) - /unlink 30000 (time 1756677383.87 total 37.97 last 805.89) - /unlink 37828 (time 1756677393.87 total 47.97 last 782.80) - /unlink 40000 (time 1756677396.53 total 50.64 last 815.96) - /unlink 47835 (time 1756677406.53 total 60.64 last 783.43) - /unlink 50000 (time 1756677409.28 total 63.38 last 788.98) - /unlink 58343 (time 1756677419.28 total 73.38 last 834.24) - /unlink 60000 (time 1756677421.45 total 75.55 last 762.98) total: 64768 /unlink in 82.18 seconds: 788.15 ops/second Waiting for MDT destroys to complete oleg119-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg119-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2