== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:14:23 (1749478463) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5382 (time 1749478504.08 total 10.00 last 538.13) - open/close 10000 (time 1749478512.92 total 18.84 last 522.47) - open/close 15178 (time 1749478522.92 total 28.84 last 517.80) - open/close 20000 (time 1749478532.08 total 38.00 last 526.54) - open/close 25498 (time 1749478542.08 total 48.00 last 549.75) - open/close 30000 (time 1749478550.17 total 56.09 last 556.17) - open/close 35448 (time 1749478560.18 total 66.09 last 544.76) - open/close 40000 (time 1749478568.76 total 74.68 last 529.99) - open/close 45443 (time 1749478578.77 total 84.69 last 544.17) - open/close 50000 (time 1749478587.89 total 93.81 last 499.42) - open/close 55414 (time 1749478597.89 total 103.81 last 541.33) - open/close 60000 (time 1749478607.34 total 113.26 last 485.45) total: 64768 open/close in 122.31 seconds: 529.54 ops/second - /unlink 9402 (time 1749478627.26 total 10.00 last 940.16) - /unlink 18994 (time 1749478637.26 total 20.00 last 959.12) - /unlink 28616 (time 1749478647.26 total 30.00 last 962.12) - /unlink 38289 (time 1749478657.26 total 40.00 last 967.19) - /unlink 48042 (time 1749478667.26 total 50.00 last 975.29) - /unlink 50000 (time 1749478669.33 total 52.07 last 946.43) - /unlink 59693 (time 1749478679.33 total 62.07 last 969.23) total: 64768 /unlink in 67.23 seconds: 963.42 ops/second Waiting for MDT destroys to complete - open/close 5571 (time 1749478788.15 total 10.00 last 557.08) - open/close 10000 (time 1749478796.19 total 18.04 last 550.78) - open/close 15637 (time 1749478806.20 total 28.04 last 563.64) - open/close 20000 (time 1749478814.11 total 35.96 last 551.12) - open/close 25471 (time 1749478824.11 total 45.96 last 547.04) - open/close 30000 (time 1749478832.37 total 54.22 last 548.54) - open/close 35396 (time 1749478842.37 total 64.22 last 539.51) - open/close 40000 (time 1749478851.15 total 73.00 last 524.49) - open/close 45383 (time 1749478861.15 total 83.00 last 538.29) - open/close 50000 (time 1749478869.71 total 91.56 last 539.44) - open/close 55387 (time 1749478879.71 total 101.56 last 538.65) - open/close 60000 (time 1749478888.08 total 109.92 last 551.31) total: 64768 open/close in 118.68 seconds: 545.73 ops/second - /unlink 9590 (time 1749478907.74 total 10.00 last 958.95) - /unlink 18447 (time 1749478917.74 total 20.00 last 885.67) - /unlink 26847 (time 1749478927.74 total 30.00 last 839.95) - /unlink 30000 (time 1749478931.12 total 33.39 last 931.87) - /unlink 39899 (time 1749478941.12 total 43.39 last 989.85) - /unlink 48102 (time 1749478951.13 total 53.39 last 820.22) - /unlink 50000 (time 1749478953.30 total 55.56 last 872.17) - /unlink 57473 (time 1749478963.30 total 65.56 last 747.27) - /unlink 60000 (time 1749478966.40 total 68.67 last 814.50) total: 64768 /unlink in 73.89 seconds: 876.50 ops/second Waiting for MDT destroys to complete - open/close 3836 (time 1749479068.98 total 10.00 last 383.58) - open/close 9150 (time 1749479078.98 total 20.00 last 531.33) - open/close 14379 (time 1749479088.98 total 30.00 last 522.89) - open/close 19015 (time 1749479098.98 total 40.00 last 463.55) - open/close 24026 (time 1749479108.98 total 50.00 last 501.07) - open/close 28593 (time 1749479118.99 total 60.00 last 456.67) - open/close 30000 (time 1749479122.08 total 63.10 last 454.28) - open/close 34652 (time 1749479132.08 total 73.10 last 465.12) - open/close 39418 (time 1749479142.09 total 83.10 last 476.53) - open/close 44722 (time 1749479152.09 total 93.11 last 530.35) - open/close 49355 (time 1749479162.09 total 103.11 last 463.24) - open/close 54511 (time 1749479172.09 total 113.11 last 515.56) - open/close 59487 (time 1749479182.09 total 123.11 last 497.47) - open/close 64619 (time 1749479192.09 total 133.11 last 513.14) total: 64768 open/close in 133.38 seconds: 485.58 ops/second - /unlink 9302 (time 1749479203.26 total 10.00 last 930.12) - /unlink 18902 (time 1749479213.26 total 20.00 last 959.99) - /unlink 28533 (time 1749479223.26 total 30.00 last 963.04) - /unlink 38333 (time 1749479233.26 total 40.00 last 979.96) - /unlink 47957 (time 1749479243.26 total 50.00 last 962.39) - /unlink 50000 (time 1749479245.35 total 52.09 last 979.95) - /unlink 59896 (time 1749479255.35 total 62.09 last 989.59) total: 64768 /unlink in 67.03 seconds: 966.21 ops/second Waiting for MDT destroys to complete - open/close 5617 (time 1749479364.21 total 10.00 last 561.66) - open/close 10000 (time 1749479371.90 total 17.69 last 570.21) - open/close 15633 (time 1749479381.90 total 27.69 last 563.25) - open/close 20000 (time 1749479389.84 total 35.63 last 549.75) - open/close 25552 (time 1749479399.84 total 45.63 last 555.12) - open/close 30000 (time 1749479407.87 total 53.66 last 554.39) - open/close 35501 (time 1749479417.87 total 63.66 last 550.04) - open/close 40000 (time 1749479426.14 total 71.94 last 543.50) - open/close 45189 (time 1749479436.15 total 81.94 last 518.84) - open/close 50000 (time 1749479445.76 total 91.55 last 500.25) - open/close 55395 (time 1749479455.76 total 101.55 last 539.45) - open/close 60000 (time 1749479464.28 total 110.07 last 540.72) total: 64768 open/close in 118.90 seconds: 544.74 ops/second - /unlink 9346 (time 1749479484.01 total 10.00 last 934.52) - /unlink 18472 (time 1749479494.01 total 20.00 last 912.54) - /unlink 28104 (time 1749479504.01 total 30.00 last 963.14) - /unlink 37850 (time 1749479514.01 total 40.00 last 974.51) - /unlink 40000 (time 1749479516.19 total 42.18 last 986.40) - /unlink 50000 (time 1749479526.07 total 52.06 last 1012.41) - /unlink 60000 (time 1749479535.82 total 61.81 last 1025.40) total: 64768 /unlink in 66.34 seconds: 976.29 ops/second Waiting for MDT destroys to complete - open/close 5624 (time 1749479643.47 total 10.00 last 562.39) - open/close 10000 (time 1749479651.31 total 17.83 last 558.59) - open/close 15557 (time 1749479661.31 total 27.83 last 555.69) - open/close 20000 (time 1749479669.17 total 35.70 last 565.20) - open/close 25521 (time 1749479679.17 total 45.70 last 552.05) - open/close 30000 (time 1749479687.54 total 54.07 last 535.02) - open/close 35151 (time 1749479697.54 total 64.07 last 515.06) - open/close 40000 (time 1749479706.41 total 72.93 last 547.01) - open/close 44827 (time 1749479716.41 total 82.93 last 482.65) - open/close 49710 (time 1749479726.41 total 92.94 last 488.22) - open/close 54904 (time 1749479736.41 total 102.94 last 519.34) - open/close 60000 (time 1749479745.96 total 112.49 last 533.63) total: 64768 open/close in 121.48 seconds: 533.18 ops/second - /unlink 9605 (time 1749479765.81 total 10.00 last 960.46) - /unlink 18935 (time 1749479775.81 total 20.00 last 933.00) - /unlink 28573 (time 1749479785.81 total 30.00 last 963.77) - /unlink 38389 (time 1749479795.81 total 40.00 last 981.55) - /unlink 48394 (time 1749479805.81 total 50.00 last 1000.49) - /unlink 58574 (time 1749479815.81 total 60.00 last 1017.92) total: 64768 /unlink in 66.08 seconds: 980.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2