== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:44:16 (1739533456) Stopping clients: oleg343-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg343-client.virtnet /mnt/lustre2 (opts:-f) oleg343-server: oleg343-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg343-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg343-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg343-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1516 (time 1739533518.37 total 10.00 last 151.57) - open/close 3014 (time 1739533528.37 total 20.01 last 149.75) - open/close 4479 (time 1739533538.38 total 30.01 last 146.47) - open/close 5944 (time 1739533548.38 total 40.01 last 146.48) - open/close 7398 (time 1739533558.38 total 50.01 last 145.38) - open/close 8909 (time 1739533568.38 total 60.01 last 151.09) - open/close 10000 (time 1739533575.49 total 67.12 last 153.41) - open/close 11468 (time 1739533585.51 total 77.14 last 146.59) - open/close 13158 (time 1739533595.51 total 87.14 last 169.00) - open/close 14998 (time 1739533605.51 total 97.14 last 184.00) - open/close 16670 (time 1739533615.51 total 107.14 last 167.10) - open/close 18889 (time 1739533625.52 total 117.15 last 221.81) - open/close 20000 (time 1739533630.92 total 122.55 last 205.78) - open/close 21636 (time 1739533640.92 total 132.55 last 163.55) - open/close 23297 (time 1739533650.92 total 142.55 last 166.06) - open/close 25096 (time 1739533660.93 total 152.56 last 179.80) - open/close 27089 (time 1739533670.93 total 162.56 last 199.25) - open/close 29203 (time 1739533680.93 total 172.56 last 211.39) - open/close 30000 (time 1739533685.56 total 177.19 last 172.26) - open/close 31859 (time 1739533695.56 total 187.19 last 185.88) - open/close 33615 (time 1739533705.56 total 197.19 last 175.57) - open/close 35225 (time 1739533715.56 total 207.19 last 160.92) - open/close 36974 (time 1739533725.57 total 217.20 last 174.83) - open/close 38800 (time 1739533735.58 total 227.21 last 182.45) - open/close 40000 (time 1739533741.25 total 232.89 last 211.28) - open/close 42141 (time 1739533751.26 total 242.89 last 214.03) - open/close 44696 (time 1739533761.26 total 252.89 last 255.47) - open/close 46601 (time 1739533771.26 total 262.89 last 190.48) - open/close 48319 (time 1739533781.27 total 272.90 last 171.69) - open/close 50000 (time 1739533790.93 total 282.56 last 173.95) - open/close 51716 (time 1739533800.94 total 292.57 last 171.46) - open/close 53526 (time 1739533810.94 total 302.57 last 180.93) - open/close 55528 (time 1739533820.94 total 312.57 last 200.17) - open/close 57377 (time 1739533830.95 total 322.58 last 184.87) - open/close 59258 (time 1739533840.95 total 332.58 last 188.05) - open/close 60000 (time 1739533844.24 total 335.87 last 225.33) - open/close 62001 (time 1739533854.25 total 345.88 last 199.91) - open/close 64025 (time 1739533864.26 total 355.89 last 202.26) total: 64768 open/close in 359.52 seconds: 180.15 ops/second - /unlink 3715 (time 1739533879.04 total 10.00 last 371.43) - /unlink 6922 (time 1739533889.04 total 20.00 last 320.69) - /unlink 10000 (time 1739533898.59 total 29.55 last 322.26) - /unlink 14426 (time 1739533908.60 total 39.56 last 442.52) - /unlink 18243 (time 1739533918.60 total 49.56 last 381.60) - /unlink 20000 (time 1739533923.60 total 54.56 last 351.48) - /unlink 23710 (time 1739533933.60 total 64.56 last 370.94) - /unlink 28544 (time 1739533943.60 total 74.56 last 483.30) - /unlink 30000 (time 1739533947.24 total 78.20 last 400.31) - /unlink 34714 (time 1739533957.24 total 88.20 last 471.39) - /unlink 40000 (time 1739533966.49 total 97.45 last 571.34) - /unlink 44983 (time 1739533976.49 total 107.45 last 498.27) - /unlink 49818 (time 1739533986.49 total 117.45 last 483.32) - /unlink 54452 (time 1739533996.50 total 127.46 last 463.30) - /unlink 59482 (time 1739534006.50 total 137.46 last 502.97) - /unlink 63916 (time 1739534016.50 total 147.46 last 443.18) total: 64768 /unlink in 149.10 seconds: 434.40 ops/second Waiting for MDT destroys to complete oleg343-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg343-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 - open/close 3955 (time 1739534039.60 total 10.01 last 395.28) - open/close 7287 (time 1739534049.60 total 20.01 last 333.16) - open/close 10000 (time 1739534056.11 total 26.51 last 416.86) - open/close 14241 (time 1739534066.11 total 36.52 last 424.08) - open/close 17393 (time 1739534076.11 total 46.52 last 315.15) - open/close 20000 (time 1739534084.52 total 54.92 last 310.17) - open/close 22872 (time 1739534094.52 total 64.92 last 287.19) - open/close 27251 (time 1739534104.52 total 74.92 last 437.83) - open/close 30000 (time 1739534110.44 total 80.84 last 464.55) - open/close 34187 (time 1739534120.44 total 90.84 last 418.64) - open/close 39193 (time 1739534130.44 total 100.84 last 500.53) - open/close 44057 (time 1739534140.44 total 110.84 last 486.40) - open/close 48186 (time 1739534150.44 total 120.85 last 412.86) - open/close 50000 (time 1739534154.75 total 125.15 last 420.98) - open/close 55240 (time 1739534164.75 total 135.15 last 523.97) - open/close 60000 (time 1739534174.49 total 144.90 last 488.51) - open/close 64243 (time 1739534184.50 total 154.90 last 424.18) total: 64768 open/close in 156.04 seconds: 415.07 ops/second - /unlink 8458 (time 1739534196.56 total 10.00 last 845.66) - /unlink 17802 (time 1739534206.56 total 20.00 last 934.39) - /unlink 20000 (time 1739534208.74 total 22.18 last 1008.65) - /unlink 28287 (time 1739534218.74 total 32.18 last 828.65) - /unlink 30000 (time 1739534221.35 total 34.80 last 654.98) - /unlink 38545 (time 1739534231.36 total 44.80 last 854.46) - /unlink 40000 (time 1739534233.43 total 46.87 last 701.00) - /unlink 46509 (time 1739534243.43 total 56.87 last 650.87) - /unlink 50000 (time 1739534247.89 total 61.34 last 782.19) - /unlink 58475 (time 1739534257.89 total 71.34 last 847.50) - /unlink 60000 (time 1739534259.92 total 73.36 last 753.41) total: 64768 /unlink in 79.06 seconds: 819.26 ops/second Waiting for MDT destroys to complete oleg343-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg343-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 - open/close 3947 (time 1739534353.42 total 10.00 last 394.56) - open/close 7806 (time 1739534363.42 total 20.01 last 385.84) - open/close 10000 (time 1739534369.45 total 26.04 last 363.69) - open/close 13116 (time 1739534379.45 total 36.04 last 311.57) - open/close 17677 (time 1739534389.45 total 46.04 last 456.06) - open/close 20000 (time 1739534393.92 total 50.51 last 520.03) - open/close 24378 (time 1739534403.92 total 60.51 last 437.75) - open/close 30000 (time 1739534413.54 total 70.13 last 584.40) - open/close 33946 (time 1739534423.54 total 80.13 last 394.59) - open/close 39277 (time 1739534433.54 total 90.13 last 533.03) - open/close 44937 (time 1739534443.54 total 100.13 last 565.95) - open/close 49601 (time 1739534453.54 total 110.13 last 466.35) - open/close 54483 (time 1739534463.54 total 120.13 last 488.16) - open/close 59258 (time 1739534473.55 total 130.13 last 477.47) - open/close 64146 (time 1739534483.55 total 140.14 last 488.69) total: 64768 open/close in 142.37 seconds: 454.94 ops/second - /unlink 8538 (time 1739534496.77 total 10.00 last 853.78) - /unlink 16633 (time 1739534506.77 total 20.00 last 809.28) - /unlink 20000 (time 1739534511.90 total 25.13 last 657.19) - /unlink 28870 (time 1739534521.90 total 35.13 last 886.93) - /unlink 34373 (time 1739534531.90 total 45.13 last 550.15) - /unlink 36964 (time 1739534541.90 total 55.13 last 259.07) - /unlink 40000 (time 1739534548.62 total 61.85 last 452.00) - /unlink 44278 (time 1739534558.62 total 71.85 last 427.75) - /unlink 49285 (time 1739534568.63 total 81.86 last 500.30) - /unlink 55862 (time 1739534578.63 total 91.86 last 657.68) - /unlink 60000 (time 1739534583.55 total 96.77 last 841.53) total: 64768 /unlink in 102.13 seconds: 634.19 ops/second Waiting for MDT destroys to complete oleg343-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg343-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 - open/close 4707 (time 1739534660.03 total 10.00 last 470.62) - open/close 8653 (time 1739534670.03 total 20.00 last 394.56) - open/close 10000 (time 1739534672.89 total 22.87 last 470.49) - open/close 15410 (time 1739534682.90 total 32.87 last 540.92) - open/close 20000 (time 1739534691.80 total 41.77 last 515.31) - open/close 24530 (time 1739534701.80 total 51.77 last 452.99) - open/close 29860 (time 1739534711.80 total 61.78 last 532.96) - open/close 34991 (time 1739534721.81 total 71.78 last 513.08) - open/close 40000 (time 1739534731.58 total 81.55 last 512.70) - open/close 44225 (time 1739534741.58 total 91.55 last 422.48) - open/close 48248 (time 1739534751.58 total 101.55 last 402.17) - open/close 49887 (time 1739534761.58 total 111.55 last 163.84) - open/close 51080 (time 1739534771.59 total 121.56 last 119.17) - open/close 52361 (time 1739534781.59 total 131.57 last 128.09) - open/close 54357 (time 1739534791.60 total 141.57 last 199.55) - open/close 56173 (time 1739534801.60 total 151.57 last 181.59) - open/close 57891 (time 1739534811.60 total 161.57 last 171.76) - open/close 59421 (time 1739534821.60 total 171.57 last 152.95) - open/close 60000 (time 1739534825.41 total 175.38 last 151.93) - open/close 61769 (time 1739534835.42 total 185.39 last 176.85) - open/close 64135 (time 1739534845.42 total 195.39 last 236.60) total: 64768 open/close in 197.83 seconds: 327.40 ops/second - /unlink 6147 (time 1739534858.92 total 10.00 last 614.67) - /unlink 10000 (time 1739534867.96 total 19.04 last 426.02) - /unlink 14215 (time 1739534877.96 total 29.05 last 421.43) - /unlink 17533 (time 1739534887.96 total 39.05 last 331.78) - /unlink 20000 (time 1739534895.53 total 46.61 last 326.03) - /unlink 22878 (time 1739534905.53 total 56.61 last 287.78) - /unlink 26125 (time 1739534915.53 total 66.62 last 324.66) - /unlink 29625 (time 1739534925.53 total 76.62 last 349.99) - /unlink 32594 (time 1739534935.53 total 86.62 last 296.88) - /unlink 36365 (time 1739534945.55 total 96.63 last 376.49) - /unlink 39794 (time 1739534955.55 total 106.63 last 342.90) - /unlink 43997 (time 1739534965.55 total 116.64 last 420.20) - /unlink 50000 (time 1739534974.99 total 126.08 last 635.83) - /unlink 56152 (time 1739534984.99 total 136.08 last 615.20) - /unlink 60000 (time 1739534992.16 total 143.24 last 537.15) total: 64768 /unlink in 151.42 seconds: 427.74 ops/second Waiting for MDT destroys to complete oleg343-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 oleg343-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 - open/close 2306 (time 1739535014.79 total 10.00 last 230.54) - open/close 4321 (time 1739535024.79 total 20.01 last 201.45) - open/close 6375 (time 1739535034.80 total 30.01 last 205.36) - open/close 8260 (time 1739535044.82 total 40.03 last 188.09) - open/close 10000 (time 1739535053.48 total 48.69 last 200.97) - open/close 12213 (time 1739535063.48 total 58.69 last 221.27) - open/close 14667 (time 1739535073.48 total 68.69 last 245.36) - open/close 17548 (time 1739535083.48 total 78.69 last 288.07) - open/close 20000 (time 1739535093.38 total 88.59 last 247.73) - open/close 22414 (time 1739535103.38 total 98.59 last 241.38) - open/close 24954 (time 1739535113.38 total 108.59 last 253.95) - open/close 27513 (time 1739535123.38 total 118.59 last 255.84) - open/close 30000 (time 1739535132.68 total 127.89 last 267.65) - open/close 32654 (time 1739535142.68 total 137.89 last 265.36) - open/close 35749 (time 1739535152.68 total 147.89 last 309.48) - open/close 38682 (time 1739535162.68 total 157.89 last 293.10) - open/close 40000 (time 1739535167.03 total 162.24 last 303.28) - open/close 43238 (time 1739535177.03 total 172.24 last 323.74) - open/close 46641 (time 1739535187.03 total 182.24 last 340.23) - open/close 49463 (time 1739535197.03 total 192.24 last 282.20) - open/close 52835 (time 1739535207.03 total 202.24 last 337.19) - open/close 55727 (time 1739535217.04 total 212.25 last 289.10) - open/close 58980 (time 1739535227.04 total 222.25 last 325.28) - open/close 60000 (time 1739535230.25 total 225.46 last 317.76) - open/close 62941 (time 1739535240.25 total 235.46 last 294.09) total: 64768 open/close in 241.36 seconds: 268.34 ops/second - /unlink 5830 (time 1739535257.21 total 10.00 last 582.96) - /unlink 10000 (time 1739535264.92 total 17.71 last 541.15) - /unlink 15260 (time 1739535274.92 total 27.71 last 525.93) - /unlink 20000 (time 1739535282.95 total 35.75 last 589.76) - /unlink 25884 (time 1739535292.95 total 45.75 last 588.39) - /unlink 30000 (time 1739535300.70 total 53.49 last 531.53) - /unlink 38602 (time 1739535310.70 total 63.49 last 860.15) - /unlink 47506 (time 1739535320.70 total 73.49 last 890.33) - /unlink 50000 (time 1739535324.86 total 77.65 last 599.86) - /unlink 57270 (time 1739535334.86 total 87.65 last 726.87) - /unlink 60000 (time 1739535338.63 total 91.42 last 724.17) total: 64768 /unlink in 97.16 seconds: 666.61 ops/second Waiting for MDT destroys to complete oleg343-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg343-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2