== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:38:47 (1773668327) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg407-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2728 (time 1773668382.06 total 10.00 last 272.71) - open/close 5541 (time 1773668392.06 total 20.00 last 281.26) - open/close 7786 (time 1773668402.06 total 30.01 last 224.48) - open/close 9783 (time 1773668412.07 total 40.01 last 199.62) - open/close 12245 (time 1773668422.07 total 50.01 last 246.18) - open/close 14540 (time 1773668432.07 total 60.01 last 229.48) - open/close 16698 (time 1773668442.07 total 70.01 last 215.77) - open/close 18967 (time 1773668452.07 total 80.01 last 226.85) - open/close 20000 (time 1773668457.33 total 85.27 last 196.57) - open/close 22292 (time 1773668467.33 total 95.27 last 229.18) - open/close 24700 (time 1773668477.33 total 105.27 last 240.75) - open/close 26847 (time 1773668487.33 total 115.27 last 214.67) - open/close 28827 (time 1773668497.34 total 125.28 last 197.86) - open/close 30000 (time 1773668503.79 total 131.74 last 181.65) - open/close 32136 (time 1773668513.80 total 141.74 last 213.52) - open/close 34934 (time 1773668523.80 total 151.75 last 279.71) - open/close 37263 (time 1773668533.81 total 161.75 last 232.80) - open/close 38864 (time 1773668543.81 total 171.76 last 160.00) - open/close 39767 (time 1773668553.82 total 181.76 last 90.23) - open/close 40000 (time 1773668556.93 total 184.88 last 74.88) - open/close 40639 (time 1773668567.02 total 194.96 last 63.34) - open/close 41344 (time 1773668577.02 total 204.97 last 70.48) - open/close 42133 (time 1773668587.03 total 214.98 last 78.83) - open/close 43062 (time 1773668597.03 total 224.98 last 92.89) - open/close 44154 (time 1773668607.03 total 234.98 last 109.19) - open/close 45261 (time 1773668617.07 total 245.02 last 110.29) - open/close 46398 (time 1773668627.10 total 255.05 last 113.32) - open/close 47468 (time 1773668637.11 total 265.05 last 106.98) - open/close 48338 (time 1773668647.14 total 275.08 last 86.72) - open/close 49302 (time 1773668657.15 total 285.09 last 96.31) - open/close 50000 (time 1773668664.02 total 291.96 last 101.63) - open/close 51096 (time 1773668674.02 total 301.96 last 109.57) - open/close 52322 (time 1773668684.02 total 311.97 last 122.57) - open/close 53458 (time 1773668694.03 total 321.97 last 113.56) - open/close 54663 (time 1773668704.03 total 331.98 last 120.44) - open/close 55839 (time 1773668714.03 total 341.98 last 117.57) - open/close 56866 (time 1773668724.03 total 351.98 last 102.70) - open/close 58240 (time 1773668734.04 total 361.98 last 137.37) - open/close 59491 (time 1773668744.04 total 371.98 last 125.05) - open/close 60000 (time 1773668747.99 total 375.94 last 128.81) - open/close 61315 (time 1773668758.00 total 385.94 last 131.42) - open/close 62544 (time 1773668768.00 total 395.94 last 122.88) - open/close 63649 (time 1773668778.01 total 405.95 last 110.43) total: 64768 open/close in 415.64 seconds: 155.83 ops/second - /unlink 1916 (time 1773668800.37 total 10.00 last 191.57) - /unlink 3878 (time 1773668810.37 total 20.00 last 196.19) - /unlink 5914 (time 1773668820.37 total 30.00 last 203.58) - /unlink 7784 (time 1773668830.37 total 40.01 last 186.95) - /unlink 9918 (time 1773668840.38 total 50.01 last 213.37) - /unlink 11902 (time 1773668850.38 total 60.01 last 198.29) - /unlink 14010 (time 1773668860.39 total 70.02 last 210.66) - /unlink 16177 (time 1773668870.39 total 80.02 last 216.59) - /unlink 18112 (time 1773668880.40 total 90.03 last 193.44) - /unlink 20000 (time 1773668889.36 total 98.99 last 210.59) - /unlink 22100 (time 1773668899.36 total 108.99 last 209.98) - /unlink 24215 (time 1773668909.37 total 119.00 last 211.41) - /unlink 26474 (time 1773668919.37 total 129.00 last 225.89) - /unlink 28605 (time 1773668929.37 total 139.00 last 213.06) - /unlink 30000 (time 1773668935.22 total 144.85 last 238.58) - /unlink 32249 (time 1773668945.22 total 154.85 last 224.88) - /unlink 34320 (time 1773668955.22 total 164.85 last 207.02) - /unlink 36370 (time 1773668965.22 total 174.85 last 204.99) - /unlink 38613 (time 1773668975.22 total 184.85 last 224.28) - /unlink 40000 (time 1773668981.56 total 191.19 last 218.92) - /unlink 42047 (time 1773668991.56 total 201.19 last 204.60) - /unlink 44763 (time 1773669001.57 total 211.20 last 271.46) - /unlink 47455 (time 1773669011.57 total 221.20 last 269.17) - /unlink 50000 (time 1773669021.00 total 230.63 last 269.93) - /unlink 52294 (time 1773669031.00 total 240.64 last 229.25) - /unlink 54845 (time 1773669041.00 total 250.64 last 255.10) - /unlink 57588 (time 1773669051.01 total 260.64 last 274.23) - /unlink 59898 (time 1773669061.01 total 270.64 last 230.96) - /unlink 62195 (time 1773669071.01 total 280.64 last 229.63) total: 64768 /unlink in 290.18 seconds: 223.20 ops/second Waiting for MDT destroys to complete oleg407-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 - open/close 1365 (time 1773669100.09 total 10.01 last 136.43) - open/close 2682 (time 1773669110.09 total 20.01 last 131.67) - open/close 4045 (time 1773669120.10 total 30.01 last 136.24) - open/close 5387 (time 1773669130.10 total 40.01 last 134.17) - open/close 6695 (time 1773669140.10 total 50.01 last 130.78) - open/close 7993 (time 1773669150.10 total 60.02 last 129.76) - open/close 9533 (time 1773669160.11 total 70.02 last 153.95) - open/close 10000 (time 1773669163.54 total 73.46 last 135.88) - open/close 11589 (time 1773669173.56 total 83.47 last 158.72) - open/close 12734 (time 1773669183.56 total 93.48 last 114.42) - open/close 13295 (time 1773669193.58 total 103.50 last 55.98) - open/close 13910 (time 1773669203.59 total 113.51 last 61.45) - open/close 14378 (time 1773669213.60 total 123.51 last 46.78) - open/close 14871 (time 1773669223.60 total 133.51 last 49.29) - open/close 15320 (time 1773669233.60 total 143.51 last 44.90) - open/close 15706 (time 1773669243.61 total 153.52 last 38.56) - open/close 16143 (time 1773669253.64 total 163.55 last 43.57) - open/close 16686 (time 1773669263.67 total 173.58 last 54.15) - open/close 17317 (time 1773669273.68 total 183.59 last 63.03) - open/close 18043 (time 1773669283.69 total 193.60 last 72.52) - open/close 18750 (time 1773669293.71 total 203.63 last 70.53) - open/close 19544 (time 1773669303.72 total 213.63 last 79.38) - open/close 20000 (time 1773669309.20 total 219.11 last 83.16) - open/close 20781 (time 1773669319.20 total 229.11 last 78.10) - open/close 21585 (time 1773669329.21 total 239.12 last 80.32) - open/close 22372 (time 1773669339.22 total 249.14 last 78.59) - open/close 23123 (time 1773669349.23 total 259.14 last 75.06) - open/close 23880 (time 1773669359.23 total 269.15 last 75.64) - open/close 24666 (time 1773669369.25 total 279.17 last 78.45) - open/close 25376 (time 1773669379.26 total 289.18 last 70.92) - open/close 26095 (time 1773669389.27 total 299.18 last 71.87) - open/close 26880 (time 1773669399.27 total 309.18 last 78.48) - open/close 27695 (time 1773669409.27 total 319.18 last 81.50) - open/close 28601 (time 1773669419.28 total 329.19 last 90.53) - open/close 29417 (time 1773669429.28 total 339.19 last 81.59) - open/close 30000 (time 1773669435.98 total 345.89 last 87.06) - open/close 30840 (time 1773669445.98 total 355.90 last 83.93) - open/close 31746 (time 1773669455.99 total 365.90 last 90.59) - open/close 32580 (time 1773669465.99 total 375.90 last 83.38) - open/close 33435 (time 1773669475.99 total 385.91 last 85.47) - open/close 34130 (time 1773669486.00 total 395.91 last 69.44) - open/close 34925 (time 1773669496.01 total 405.92 last 79.45) - open/close 35757 (time 1773669506.01 total 415.93 last 83.14) - open/close 36602 (time 1773669516.02 total 425.93 last 84.48) - open/close 37362 (time 1773669526.02 total 435.93 last 75.95) - open/close 38158 (time 1773669536.02 total 445.94 last 79.58) - open/close 38926 (time 1773669546.03 total 455.95 last 76.74) - open/close 39751 (time 1773669556.05 total 465.96 last 82.37) - open/close 40000 (time 1773669559.43 total 469.34 last 73.70) - open/close 40806 (time 1773669569.44 total 479.35 last 80.52) - open/close 41673 (time 1773669579.44 total 489.36 last 86.66) - open/close 42599 (time 1773669589.44 total 499.36 last 92.59) - open/close 43516 (time 1773669599.45 total 509.36 last 91.64) - open/close 44491 (time 1773669609.45 total 519.37 last 97.45) - open/close 45445 (time 1773669619.46 total 529.37 last 95.38) - open/close 46415 (time 1773669629.46 total 539.38 last 96.93) - open/close 47440 (time 1773669639.47 total 549.38 last 102.47) - open/close 48546 (time 1773669649.47 total 559.38 last 110.57) - open/close 49808 (time 1773669659.47 total 569.39 last 126.17) - open/close 50960 (time 1773669669.49 total 579.40 last 115.04) - open/close 52345 (time 1773669679.49 total 589.40 last 138.46) - open/close 53573 (time 1773669689.49 total 599.40 last 122.77) - open/close 54915 (time 1773669699.49 total 609.41 last 134.17) - open/close 56278 (time 1773669709.50 total 619.42 last 136.18) - open/close 57403 (time 1773669719.51 total 629.42 last 112.44) - open/close 58710 (time 1773669729.51 total 639.42 last 130.69) - open/close 60000 (time 1773669738.06 total 647.97 last 150.92) - open/close 61473 (time 1773669748.06 total 657.97 last 147.27) - open/close 62895 (time 1773669758.06 total 667.98 last 142.13) - open/close 64418 (time 1773669768.07 total 677.98 last 152.25) total: 64768 open/close in 680.66 seconds: 95.15 ops/second - /unlink 1546 (time 1773669783.06 total 10.00 last 154.57) - /unlink 2818 (time 1773669793.06 total 20.01 last 127.15) - /unlink 4491 (time 1773669803.06 total 30.01 last 167.26) - /unlink 6219 (time 1773669813.07 total 40.01 last 172.72) - /unlink 8698 (time 1773669823.07 total 50.01 last 247.84) - /unlink 10000 (time 1773669828.15 total 55.10 last 256.20) - /unlink 12676 (time 1773669838.15 total 65.10 last 267.56) - /unlink 16215 (time 1773669848.15 total 75.10 last 353.87) - /unlink 20000 (time 1773669856.33 total 83.27 last 463.01) - /unlink 25172 (time 1773669866.33 total 93.28 last 517.00) - /unlink 30000 (time 1773669874.49 total 101.43 last 591.96) - /unlink 35640 (time 1773669884.49 total 111.43 last 564.00) - /unlink 40000 (time 1773669890.41 total 117.35 last 736.58) - /unlink 45352 (time 1773669900.41 total 127.35 last 535.17) - /unlink 50000 (time 1773669909.41 total 136.36 last 516.06) - /unlink 56415 (time 1773669919.42 total 146.36 last 641.41) - /unlink 60000 (time 1773669925.54 total 152.49 last 585.37) total: 64768 /unlink in 160.56 seconds: 403.38 ops/second Waiting for MDT destroys to complete oleg407-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 - open/close 2452 (time 1773669953.93 total 10.00 last 245.18) - open/close 4712 (time 1773669963.94 total 20.00 last 225.90) - open/close 7789 (time 1773669973.94 total 30.01 last 307.68) - open/close 10000 (time 1773669980.84 total 36.91 last 320.23) - open/close 12806 (time 1773669990.85 total 46.91 last 280.57) - open/close 16154 (time 1773670000.85 total 56.91 last 334.78) - open/close 19001 (time 1773670010.85 total 66.91 last 284.66) - open/close 20000 (time 1773670014.45 total 70.51 last 277.55) - open/close 22060 (time 1773670024.45 total 80.52 last 205.94) - open/close 23951 (time 1773670034.45 total 90.52 last 189.08) - open/close 26398 (time 1773670044.45 total 100.52 last 244.65) - open/close 29250 (time 1773670054.45 total 110.52 last 285.19) - open/close 30000 (time 1773670057.06 total 113.12 last 288.12) - open/close 32655 (time 1773670067.06 total 123.12 last 265.47) - open/close 35438 (time 1773670077.06 total 133.12 last 278.30) - open/close 38180 (time 1773670087.06 total 143.13 last 274.14) - open/close 40000 (time 1773670093.09 total 149.16 last 301.66) - open/close 43159 (time 1773670103.10 total 159.16 last 315.74) - open/close 45877 (time 1773670113.11 total 169.17 last 271.56) - open/close 48774 (time 1773670123.11 total 179.18 last 289.63) - open/close 50000 (time 1773670126.89 total 182.95 last 324.40) - open/close 52859 (time 1773670136.89 total 192.96 last 285.74) - open/close 55349 (time 1773670146.90 total 202.96 last 248.97) - open/close 58390 (time 1773670156.90 total 212.96 last 304.06) - open/close 60000 (time 1773670162.17 total 218.23 last 305.47) - open/close 63038 (time 1773670172.17 total 228.23 last 303.79) total: 64768 open/close in 234.42 seconds: 276.29 ops/second - /unlink 4949 (time 1773670189.92 total 10.00 last 494.85) - /unlink 9005 (time 1773670199.95 total 20.03 last 404.55) - /unlink 10000 (time 1773670202.45 total 22.54 last 396.50) - /unlink 14360 (time 1773670212.46 total 32.54 last 435.98) - /unlink 19563 (time 1773670222.46 total 42.54 last 520.28) - /unlink 26012 (time 1773670232.46 total 52.54 last 644.82) - /unlink 30000 (time 1773670238.50 total 58.58 last 659.71) - /unlink 37021 (time 1773670248.50 total 68.58 last 702.01) - /unlink 40000 (time 1773670253.35 total 73.43 last 614.34) - /unlink 46552 (time 1773670263.35 total 83.43 last 655.16) - /unlink 50000 (time 1773670268.62 total 88.70 last 654.89) - /unlink 56003 (time 1773670278.62 total 98.70 last 600.18) - /unlink 60000 (time 1773670287.53 total 107.61 last 448.66) total: 64768 /unlink in 117.12 seconds: 552.99 ops/second Waiting for MDT destroys to complete oleg407-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 - open/close 3726 (time 1773670316.74 total 10.00 last 372.51) - open/close 7684 (time 1773670326.74 total 20.00 last 395.72) - open/close 10000 (time 1773670332.50 total 25.77 last 401.89) - open/close 13728 (time 1773670342.51 total 35.77 last 372.73) - open/close 18049 (time 1773670352.51 total 45.77 last 432.03) - open/close 20000 (time 1773670357.80 total 51.06 last 368.86) - open/close 23359 (time 1773670367.80 total 61.06 last 335.83) - open/close 27257 (time 1773670377.80 total 71.06 last 389.69) - open/close 30000 (time 1773670385.70 total 78.96 last 347.35) - open/close 33488 (time 1773670395.70 total 88.96 last 348.77) - open/close 37164 (time 1773670405.70 total 98.96 last 367.55) - open/close 40000 (time 1773670412.35 total 105.62 last 426.37) - open/close 44099 (time 1773670422.36 total 115.62 last 409.79) - open/close 48813 (time 1773670432.36 total 125.62 last 471.34) - open/close 50000 (time 1773670434.97 total 128.23 last 454.46) - open/close 54423 (time 1773670444.97 total 138.23 last 442.30) - open/close 58908 (time 1773670454.97 total 148.23 last 448.41) - open/close 60000 (time 1773670457.60 total 150.87 last 414.84) - open/close 64648 (time 1773670467.61 total 160.87 last 464.72) total: 64768 open/close in 161.08 seconds: 402.07 ops/second - /unlink 8349 (time 1773670479.11 total 10.00 last 834.89) - /unlink 10000 (time 1773670481.12 total 12.01 last 822.68) - /unlink 17978 (time 1773670491.12 total 22.01 last 797.76) - /unlink 20000 (time 1773670493.54 total 24.43 last 833.63) - /unlink 28325 (time 1773670503.54 total 34.43 last 832.47) - /unlink 37274 (time 1773670513.54 total 44.43 last 894.84) - /unlink 40000 (time 1773670516.93 total 47.82 last 804.81) - /unlink 48090 (time 1773670526.93 total 57.82 last 808.92) - /unlink 50000 (time 1773670529.08 total 59.97 last 889.39) - /unlink 58554 (time 1773670539.08 total 69.97 last 855.28) total: 64768 /unlink in 77.04 seconds: 840.69 ops/second Waiting for MDT destroys to complete oleg407-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 - open/close 5306 (time 1773670564.01 total 10.00 last 530.50) - open/close 10000 (time 1773670572.90 total 18.89 last 528.21) - open/close 15482 (time 1773670582.90 total 28.89 last 548.09) - open/close 20000 (time 1773670591.41 total 37.40 last 531.12) - open/close 25144 (time 1773670601.41 total 47.40 last 514.32) - open/close 30000 (time 1773670610.54 total 56.53 last 531.94) - open/close 34994 (time 1773670620.54 total 66.53 last 499.37) - open/close 40000 (time 1773670630.02 total 76.01 last 527.73) - open/close 45395 (time 1773670640.02 total 86.01 last 539.48) - open/close 50000 (time 1773670649.20 total 95.19 last 501.89) - open/close 55108 (time 1773670659.20 total 105.19 last 510.79) - open/close 59797 (time 1773670669.20 total 115.19 last 468.82) total: 64768 open/close in 124.80 seconds: 518.99 ops/second - /unlink 8457 (time 1773670690.10 total 10.00 last 845.66) - /unlink 10000 (time 1773670692.10 total 12.01 last 769.44) - /unlink 18069 (time 1773670702.11 total 22.01 last 806.85) - /unlink 20000 (time 1773670704.55 total 24.45 last 791.15) - /unlink 28364 (time 1773670714.55 total 34.45 last 836.32) - /unlink 36568 (time 1773670724.55 total 44.45 last 820.32) - /unlink 40000 (time 1773670728.30 total 48.21 last 913.47) - /unlink 49139 (time 1773670738.31 total 58.21 last 913.83) - /unlink 58272 (time 1773670748.31 total 68.21 last 913.23) - /unlink 60000 (time 1773670750.41 total 70.31 last 820.92) total: 64768 /unlink in 75.75 seconds: 854.99 ops/second Waiting for MDT destroys to complete oleg407-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2