== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:53:26 (1743508406) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg414-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4471 (time 1743508447.43 total 10.00 last 447.06) - open/close 9373 (time 1743508457.43 total 20.00 last 490.13) - open/close 13989 (time 1743508467.43 total 30.00 last 461.59) - open/close 19091 (time 1743508477.43 total 40.00 last 510.12) - open/close 20000 (time 1743508479.50 total 42.08 last 437.98) - open/close 24307 (time 1743508489.50 total 52.08 last 430.67) - open/close 29187 (time 1743508499.51 total 62.08 last 487.97) - open/close 34831 (time 1743508509.51 total 72.08 last 564.37) - open/close 40000 (time 1743508518.49 total 81.07 last 575.26) - open/close 45215 (time 1743508528.49 total 91.07 last 521.47) - open/close 50000 (time 1743508537.78 total 100.36 last 515.10) - open/close 54982 (time 1743508547.78 total 110.36 last 498.19) - open/close 60000 (time 1743508557.72 total 120.30 last 504.78) total: 64768 open/close in 130.06 seconds: 498.00 ops/second - /unlink 8668 (time 1743508578.36 total 10.00 last 866.63) - /unlink 16953 (time 1743508588.36 total 20.00 last 828.49) - /unlink 20000 (time 1743508591.88 total 23.52 last 866.11) - /unlink 29176 (time 1743508601.88 total 33.52 last 917.57) - /unlink 38223 (time 1743508611.88 total 43.52 last 904.62) - /unlink 46821 (time 1743508621.88 total 53.52 last 859.71) - /unlink 50000 (time 1743508625.49 total 57.13 last 881.26) - /unlink 59213 (time 1743508635.49 total 67.13 last 921.26) total: 64768 /unlink in 72.88 seconds: 888.75 ops/second Waiting for MDT destroys to complete - open/close 6354 (time 1743508741.25 total 10.00 last 635.35) - open/close 10000 (time 1743508747.22 total 15.97 last 611.18) - open/close 15879 (time 1743508757.22 total 25.97 last 587.90) - open/close 20000 (time 1743508763.86 total 32.60 last 620.89) - open/close 25800 (time 1743508773.86 total 42.61 last 579.88) - open/close 30000 (time 1743508781.94 total 50.68 last 519.90) - open/close 36005 (time 1743508791.94 total 60.69 last 600.42) - open/close 40000 (time 1743508798.82 total 67.57 last 580.67) - open/close 45586 (time 1743508808.82 total 77.57 last 558.55) - open/close 50000 (time 1743508816.22 total 84.96 last 596.85) - open/close 56128 (time 1743508826.22 total 94.96 last 612.71) - open/close 60000 (time 1743508832.88 total 101.63 last 581.07) total: 64768 open/close in 110.13 seconds: 588.10 ops/second - /unlink 10000 (time 1743508851.92 total 9.69 last 1031.72) - /unlink 19558 (time 1743508861.92 total 19.69 last 955.74) - /unlink 29935 (time 1743508871.92 total 29.69 last 1037.64) - /unlink 40000 (time 1743508880.51 total 38.29 last 1171.44) - /unlink 50000 (time 1743508889.56 total 47.33 last 1105.18) - /unlink 60000 (time 1743508898.01 total 55.78 last 1183.54) total: 64768 /unlink in 59.88 seconds: 1081.58 ops/second Waiting for MDT destroys to complete - open/close 6529 (time 1743509012.28 total 10.00 last 652.85) - open/close 10000 (time 1743509017.47 total 15.19 last 668.58) - open/close 16381 (time 1743509027.47 total 25.19 last 638.03) - open/close 20000 (time 1743509033.62 total 31.35 last 588.16) - open/close 25836 (time 1743509043.62 total 41.35 last 583.59) - open/close 30000 (time 1743509051.64 total 49.36 last 519.64) - open/close 36012 (time 1743509061.64 total 59.36 last 601.11) - open/close 40000 (time 1743509068.00 total 65.73 last 626.35) - open/close 46301 (time 1743509078.00 total 75.73 last 630.09) - open/close 50000 (time 1743509084.91 total 82.64 last 535.50) - open/close 55891 (time 1743509094.91 total 92.64 last 589.04) - open/close 60000 (time 1743509102.11 total 99.84 last 570.74) total: 64768 open/close in 108.69 seconds: 595.89 ops/second - /unlink 8576 (time 1743509121.85 total 10.00 last 857.59) - /unlink 18054 (time 1743509131.85 total 20.00 last 947.76) - /unlink 28870 (time 1743509141.85 total 30.00 last 1081.60) - /unlink 38862 (time 1743509151.85 total 40.00 last 999.20) - /unlink 48732 (time 1743509161.85 total 50.00 last 986.95) - /unlink 58636 (time 1743509171.85 total 60.00 last 990.38) total: 64768 /unlink in 66.22 seconds: 978.13 ops/second Waiting for MDT destroys to complete - open/close 5654 (time 1743509281.84 total 10.00 last 565.36) - open/close 10000 (time 1743509289.53 total 17.69 last 565.40) - open/close 15386 (time 1743509299.53 total 27.69 last 538.55) - open/close 20000 (time 1743509307.80 total 35.96 last 558.05) - open/close 25698 (time 1743509317.80 total 45.96 last 569.73) - open/close 30000 (time 1743509325.66 total 53.82 last 546.85) - open/close 35649 (time 1743509335.67 total 63.83 last 564.84) - open/close 40000 (time 1743509343.63 total 71.79 last 546.20) - open/close 45469 (time 1743509353.63 total 81.79 last 546.84) - open/close 50000 (time 1743509361.99 total 90.15 last 542.12) - open/close 55612 (time 1743509371.99 total 100.15 last 561.16) - open/close 60000 (time 1743509379.90 total 108.06 last 555.05) total: 64768 open/close in 116.66 seconds: 555.19 ops/second - /unlink 9873 (time 1743509399.38 total 10.00 last 987.27) - /unlink 20000 (time 1743509409.33 total 19.95 last 1017.54) - /unlink 30000 (time 1743509419.28 total 29.91 last 1004.51) - /unlink 40000 (time 1743509429.24 total 39.86 last 1004.35) - /unlink 50000 (time 1743509439.15 total 49.78 last 1008.78) - /unlink 59907 (time 1743509449.15 total 59.78 last 990.67) total: 64768 /unlink in 64.78 seconds: 999.77 ops/second Waiting for MDT destroys to complete - open/close 5229 (time 1743509562.80 total 10.00 last 522.87) - open/close 10000 (time 1743509572.05 total 19.26 last 515.49) - open/close 15347 (time 1743509582.05 total 29.26 last 534.67) - open/close 20000 (time 1743509590.57 total 37.77 last 546.35) - open/close 25568 (time 1743509600.57 total 47.77 last 556.77) - open/close 30000 (time 1743509608.74 total 55.94 last 542.51) - open/close 35382 (time 1743509618.74 total 65.94 last 538.16) - open/close 40000 (time 1743509627.63 total 74.83 last 519.61) - open/close 45201 (time 1743509637.63 total 84.83 last 519.96) - open/close 50000 (time 1743509647.16 total 94.36 last 503.76) - open/close 55262 (time 1743509657.16 total 104.36 last 526.20) - open/close 60000 (time 1743509665.85 total 113.06 last 544.84) total: 64768 open/close in 122.22 seconds: 529.94 ops/second - /unlink 7807 (time 1743509685.83 total 10.00 last 780.64) - /unlink 10000 (time 1743509688.07 total 12.24 last 980.81) - /unlink 19457 (time 1743509698.07 total 22.24 last 945.69) - /unlink 27226 (time 1743509708.07 total 32.24 last 776.85) - /unlink 30000 (time 1743509711.05 total 35.22 last 929.99) - /unlink 39966 (time 1743509721.05 total 45.22 last 996.55) - /unlink 48806 (time 1743509731.05 total 55.22 last 883.98) - /unlink 57097 (time 1743509741.05 total 65.22 last 829.00) - /unlink 60000 (time 1743509745.38 total 69.55 last 670.89) total: 64768 /unlink in 75.34 seconds: 859.68 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2