== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:03:15 (1738479795) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg332-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 6331 (time 1738479835.08 total 10.00 last 633.06) - open/close 10000 (time 1738479841.92 total 16.84 last 536.50) - open/close 15592 (time 1738479851.92 total 26.84 last 559.12) - open/close 20000 (time 1738479859.18 total 34.10 last 607.34) - open/close 25878 (time 1738479869.18 total 44.10 last 587.76) - open/close 30000 (time 1738479877.05 total 51.96 last 524.12) - open/close 35516 (time 1738479887.05 total 61.97 last 551.55) - open/close 40000 (time 1738479895.53 total 70.45 last 528.52) - open/close 45208 (time 1738479905.53 total 80.45 last 520.70) - open/close 50000 (time 1738479915.00 total 89.92 last 506.29) - open/close 55005 (time 1738479925.00 total 99.92 last 500.47) - open/close 59935 (time 1738479935.00 total 109.92 last 492.98) total: 64768 open/close in 119.49 seconds: 542.04 ops/second - /unlink 9099 (time 1738479955.49 total 10.00 last 909.88) - /unlink 18246 (time 1738479965.49 total 20.00 last 914.63) - /unlink 27517 (time 1738479975.49 total 30.00 last 927.04) - /unlink 30000 (time 1738479978.19 total 32.71 last 917.96) - /unlink 39181 (time 1738479988.19 total 42.71 last 918.06) - /unlink 48566 (time 1738479998.20 total 52.71 last 938.47) - /unlink 57933 (time 1738480008.20 total 62.71 last 936.63) - /unlink 60000 (time 1738480010.53 total 65.05 last 884.43) total: 64768 /unlink in 70.38 seconds: 920.30 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 5556 (time 1738480109.15 total 10.00 last 555.51) - open/close 10000 (time 1738480117.13 total 17.98 last 556.77) - open/close 15280 (time 1738480127.14 total 27.98 last 527.92) - open/close 20000 (time 1738480135.77 total 36.62 last 546.72) - open/close 25406 (time 1738480145.77 total 46.62 last 540.57) - open/close 30000 (time 1738480154.23 total 55.07 last 543.27) - open/close 35320 (time 1738480164.23 total 65.08 last 531.94) - open/close 40000 (time 1738480172.93 total 73.78 last 537.82) - open/close 45495 (time 1738480182.93 total 83.78 last 549.47) - open/close 50000 (time 1738480191.27 total 92.12 last 540.04) - open/close 55460 (time 1738480201.27 total 102.12 last 545.99) - open/close 60000 (time 1738480209.69 total 110.54 last 539.27) total: 64768 open/close in 119.19 seconds: 543.42 ops/second - /unlink 9533 (time 1738480229.20 total 10.00 last 953.21) - /unlink 19205 (time 1738480239.20 total 20.00 last 967.12) - /unlink 28433 (time 1738480249.20 total 30.00 last 922.79) - /unlink 38256 (time 1738480259.20 total 40.00 last 982.24) - /unlink 48114 (time 1738480269.20 total 50.00 last 985.70) - /unlink 58176 (time 1738480279.20 total 60.00 last 1006.12) total: 64768 /unlink in 66.59 seconds: 972.60 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 5754 (time 1738480390.59 total 10.00 last 575.34) - open/close 10000 (time 1738480398.02 total 17.43 last 571.32) - open/close 15606 (time 1738480408.02 total 27.43 last 560.58) - open/close 20000 (time 1738480415.94 total 35.35 last 554.93) - open/close 25549 (time 1738480425.94 total 45.35 last 554.85) - open/close 30000 (time 1738480433.99 total 53.40 last 552.95) - open/close 35213 (time 1738480443.99 total 63.40 last 521.30) - open/close 40000 (time 1738480452.75 total 72.16 last 546.36) - open/close 45345 (time 1738480462.75 total 82.16 last 534.43) - open/close 50000 (time 1738480471.29 total 90.71 last 544.93) - open/close 55544 (time 1738480481.30 total 100.71 last 554.37) - open/close 60000 (time 1738480489.89 total 109.30 last 518.53) - open/close 64286 (time 1738480499.89 total 119.30 last 428.58) total: 64768 open/close in 120.20 seconds: 538.85 ops/second - /unlink 9562 (time 1738480511.68 total 10.00 last 956.14) - /unlink 18949 (time 1738480521.68 total 20.00 last 938.66) - /unlink 28497 (time 1738480531.68 total 30.00 last 954.78) - /unlink 36663 (time 1738480541.68 total 40.00 last 816.60) - /unlink 40000 (time 1738480546.50 total 44.81 last 693.49) - /unlink 47273 (time 1738480556.50 total 54.81 last 727.23) - /unlink 50000 (time 1738480559.99 total 58.30 last 781.35) - /unlink 57496 (time 1738480569.99 total 68.31 last 749.49) - /unlink 60000 (time 1738480574.52 total 72.84 last 552.37) total: 64768 /unlink in 79.13 seconds: 818.55 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 5002 (time 1738480664.85 total 10.00 last 500.11) - open/close 9764 (time 1738480674.85 total 20.00 last 476.16) - open/close 13261 (time 1738480684.85 total 30.00 last 349.66) - open/close 17437 (time 1738480694.85 total 40.00 last 417.56) - open/close 20000 (time 1738480701.23 total 46.38 last 402.15) - open/close 24622 (time 1738480711.23 total 56.38 last 462.15) - open/close 29208 (time 1738480721.23 total 66.38 last 458.49) - open/close 33761 (time 1738480731.23 total 76.38 last 455.27) - open/close 38474 (time 1738480741.23 total 86.38 last 471.22) - open/close 40000 (time 1738480744.65 total 89.80 last 446.41) - open/close 44423 (time 1738480754.65 total 99.80 last 442.26) - open/close 48807 (time 1738480764.65 total 109.81 last 438.30) - open/close 50000 (time 1738480767.42 total 112.57 last 430.75) - open/close 54124 (time 1738480777.42 total 122.58 last 412.38) - open/close 57587 (time 1738480787.43 total 132.58 last 346.18) - open/close 60000 (time 1738480793.57 total 138.73 last 392.51) - open/close 63224 (time 1738480803.58 total 148.73 last 322.33) total: 64768 open/close in 154.72 seconds: 418.61 ops/second - /unlink 5041 (time 1738480820.82 total 10.00 last 504.04) - /unlink 9680 (time 1738480830.82 total 20.00 last 463.81) - /unlink 14456 (time 1738480840.82 total 30.00 last 477.52) - /unlink 20000 (time 1738480849.16 total 38.35 last 664.64) - /unlink 26977 (time 1738480859.17 total 48.35 last 697.66) - /unlink 30000 (time 1738480863.22 total 52.40 last 745.54) - /unlink 37234 (time 1738480873.22 total 62.40 last 723.31) - /unlink 40000 (time 1738480878.11 total 67.29 last 565.43) - /unlink 46767 (time 1738480888.11 total 77.30 last 676.64) - /unlink 50000 (time 1738480892.05 total 81.23 last 821.80) - /unlink 57459 (time 1738480902.05 total 91.23 last 745.89) - /unlink 60000 (time 1738480905.63 total 94.82 last 708.48) total: 64768 /unlink in 101.73 seconds: 636.69 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 - open/close 3796 (time 1738480966.20 total 10.00 last 379.54) - open/close 8427 (time 1738480976.20 total 20.00 last 463.09) - open/close 10000 (time 1738480979.89 total 23.69 last 426.27) - open/close 14345 (time 1738480989.89 total 33.69 last 434.42) - open/close 18831 (time 1738480999.89 total 43.69 last 448.60) - open/close 20000 (time 1738481002.09 total 45.89 last 531.37) - open/close 24964 (time 1738481012.10 total 55.89 last 496.35) - open/close 29748 (time 1738481022.10 total 65.89 last 478.39) - open/close 34249 (time 1738481032.10 total 75.90 last 450.01) - open/close 38517 (time 1738481042.10 total 85.90 last 426.79) - open/close 40000 (time 1738481045.01 total 88.81 last 509.52) - open/close 44522 (time 1738481055.01 total 98.81 last 452.17) - open/close 49302 (time 1738481065.01 total 108.81 last 477.92) - open/close 54183 (time 1738481075.01 total 118.81 last 488.05) - open/close 59373 (time 1738481085.01 total 128.81 last 518.96) - open/close 64307 (time 1738481095.01 total 138.81 last 493.36) total: 64768 open/close in 139.72 seconds: 463.56 ops/second - /unlink 9097 (time 1738481106.77 total 10.00 last 909.62) - /unlink 18037 (time 1738481116.77 total 20.00 last 893.99) - /unlink 20000 (time 1738481118.95 total 22.18 last 901.00) - /unlink 29396 (time 1738481128.95 total 32.18 last 939.58) - /unlink 38736 (time 1738481138.95 total 42.18 last 933.96) - /unlink 48140 (time 1738481148.95 total 52.18 last 940.33) - /unlink 50000 (time 1738481151.02 total 54.25 last 898.34) - /unlink 58007 (time 1738481161.03 total 64.25 last 800.58) - /unlink 60000 (time 1738481163.62 total 66.85 last 767.69) total: 64768 /unlink in 71.83 seconds: 901.62 ops/second Waiting for MDT destroys to complete oleg332-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg332-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2