== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:34:49 (1749540889) Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) oleg453-server: oleg453-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg453-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg453-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5085 (time 1749540930.99 total 10.00 last 508.43) - open/close 10000 (time 1749540939.64 total 18.66 last 567.75) - open/close 15379 (time 1749540949.65 total 28.66 last 537.83) - open/close 19981 (time 1749540959.65 total 38.66 last 460.04) - open/close 23522 (time 1749540969.65 total 48.67 last 353.98) - open/close 27588 (time 1749540979.65 total 58.67 last 406.51) - open/close 30000 (time 1749540985.85 total 64.86 last 389.42) - open/close 34050 (time 1749540995.85 total 74.86 last 404.97) - open/close 37912 (time 1749541005.85 total 84.86 last 386.19) - open/close 40000 (time 1749541011.70 total 90.72 last 356.78) - open/close 43444 (time 1749541021.70 total 100.72 last 344.32) - open/close 47531 (time 1749541031.70 total 110.72 last 408.69) - open/close 50000 (time 1749541038.86 total 117.88 last 344.97) - open/close 53966 (time 1749541048.86 total 127.88 last 396.56) - open/close 58187 (time 1749541058.86 total 137.88 last 422.06) - open/close 60000 (time 1749541064.40 total 143.41 last 327.53) - open/close 63607 (time 1749541074.40 total 153.41 last 360.67) total: 64768 open/close in 157.40 seconds: 411.49 ops/second - /unlink 6060 (time 1749541089.37 total 10.00 last 605.97) - /unlink 10000 (time 1749541095.32 total 15.95 last 662.00) - /unlink 16737 (time 1749541105.32 total 25.95 last 673.69) - /unlink 20000 (time 1749541110.22 total 30.85 last 666.51) - /unlink 26809 (time 1749541120.22 total 40.85 last 680.87) - /unlink 30000 (time 1749541125.15 total 45.79 last 646.31) - /unlink 36898 (time 1749541135.15 total 55.79 last 689.72) - /unlink 40000 (time 1749541139.27 total 59.90 last 753.32) - /unlink 46457 (time 1749541149.27 total 69.90 last 645.70) - /unlink 50000 (time 1749541153.73 total 74.37 last 794.34) - /unlink 58480 (time 1749541163.73 total 84.37 last 847.97) - /unlink 60000 (time 1749541165.94 total 86.58 last 687.74) total: 64768 /unlink in 92.62 seconds: 699.32 ops/second Waiting for MDT destroys to complete - open/close 3551 (time 1749541229.22 total 10.00 last 354.99) - open/close 7857 (time 1749541239.22 total 20.00 last 430.60) - open/close 10000 (time 1749541244.64 total 25.42 last 395.85) - open/close 13822 (time 1749541254.64 total 35.42 last 382.13) - open/close 17472 (time 1749541264.65 total 45.42 last 364.83) - open/close 20000 (time 1749541271.13 total 51.91 last 389.88) - open/close 23504 (time 1749541281.13 total 61.91 last 350.37) - open/close 26974 (time 1749541291.13 total 71.91 last 346.99) - open/close 30000 (time 1749541298.43 total 79.21 last 414.33) - open/close 33566 (time 1749541308.44 total 89.21 last 356.52) - open/close 37024 (time 1749541318.44 total 99.22 last 345.74) - open/close 40000 (time 1749541327.14 total 107.92 last 342.00) - open/close 43249 (time 1749541337.14 total 117.92 last 324.87) - open/close 46825 (time 1749541347.14 total 127.92 last 357.55) - open/close 50000 (time 1749541355.62 total 136.40 last 374.53) - open/close 53305 (time 1749541365.62 total 146.40 last 330.42) - open/close 56994 (time 1749541375.62 total 156.40 last 368.87) - open/close 60000 (time 1749541382.74 total 163.52 last 422.28) - open/close 64098 (time 1749541392.74 total 173.52 last 409.72) total: 64768 open/close in 175.36 seconds: 369.35 ops/second - /unlink 6652 (time 1749541405.51 total 10.00 last 665.15) - /unlink 10000 (time 1749541410.97 total 15.46 last 613.38) - /unlink 16784 (time 1749541420.97 total 25.46 last 678.34) - /unlink 20000 (time 1749541425.75 total 30.23 last 673.68) - /unlink 27160 (time 1749541435.75 total 40.23 last 715.97) - /unlink 30000 (time 1749541439.77 total 44.25 last 706.94) - /unlink 37151 (time 1749541449.77 total 54.25 last 715.09) - /unlink 40000 (time 1749541453.90 total 58.39 last 688.74) - /unlink 48423 (time 1749541463.90 total 68.39 last 842.26) - /unlink 58024 (time 1749541473.90 total 78.39 last 960.01) total: 64768 /unlink in 84.68 seconds: 764.86 ops/second Waiting for MDT destroys to complete - open/close 5457 (time 1749541560.00 total 10.00 last 545.69) - open/close 10000 (time 1749541568.22 total 18.22 last 552.48) - open/close 15056 (time 1749541578.23 total 28.22 last 505.59) - open/close 20000 (time 1749541587.26 total 37.25 last 547.42) - open/close 25394 (time 1749541597.26 total 47.25 last 539.39) - open/close 30000 (time 1749541605.85 total 55.85 last 536.13) - open/close 35522 (time 1749541615.85 total 65.85 last 552.15) - open/close 40000 (time 1749541624.47 total 74.47 last 519.34) - open/close 45521 (time 1749541634.47 total 84.47 last 552.02) - open/close 50000 (time 1749541642.53 total 92.53 last 555.68) - open/close 55292 (time 1749541652.53 total 102.53 last 529.13) - open/close 60000 (time 1749541661.28 total 111.28 last 538.33) total: 64768 open/close in 120.10 seconds: 539.27 ops/second - /unlink 9505 (time 1749541680.99 total 10.00 last 950.44) - /unlink 18778 (time 1749541690.99 total 20.00 last 927.24) - /unlink 28359 (time 1749541700.99 total 30.00 last 958.09) - /unlink 37681 (time 1749541710.99 total 40.00 last 932.15) - /unlink 40000 (time 1749541713.51 total 42.52 last 920.13) - /unlink 49222 (time 1749541723.51 total 52.52 last 922.16) - /unlink 58843 (time 1749541733.51 total 62.52 last 962.03) total: 64768 /unlink in 68.61 seconds: 943.95 ops/second Waiting for MDT destroys to complete - open/close 5456 (time 1749541825.76 total 10.00 last 545.54) - open/close 10000 (time 1749541834.02 total 18.26 last 550.26) - open/close 15291 (time 1749541844.02 total 28.26 last 529.09) - open/close 20000 (time 1749541852.73 total 36.97 last 540.51) - open/close 25035 (time 1749541862.73 total 46.97 last 503.45) - open/close 30000 (time 1749541872.00 total 56.24 last 535.68) - open/close 35051 (time 1749541882.00 total 66.24 last 505.07) - open/close 40000 (time 1749541890.96 total 75.20 last 552.36) - open/close 45484 (time 1749541900.96 total 85.20 last 548.38) - open/close 50000 (time 1749541910.54 total 94.78 last 471.33) - open/close 54306 (time 1749541920.54 total 104.78 last 430.59) - open/close 58455 (time 1749541930.54 total 114.79 last 414.81) - open/close 60000 (time 1749541933.74 total 117.98 last 482.97) - open/close 64673 (time 1749541943.74 total 127.99 last 467.26) total: 64768 open/close in 128.17 seconds: 505.34 ops/second - /unlink 6883 (time 1749541954.80 total 10.00 last 688.27) - /unlink 10000 (time 1749541958.26 total 13.46 last 900.68) - /unlink 19757 (time 1749541968.26 total 23.46 last 975.69) - /unlink 28813 (time 1749541978.26 total 33.46 last 905.53) - /unlink 36201 (time 1749541988.26 total 43.46 last 738.76) - /unlink 40000 (time 1749541992.79 total 47.99 last 839.37) - /unlink 49728 (time 1749542002.79 total 57.99 last 972.74) - /unlink 57393 (time 1749542012.79 total 67.99 last 766.45) - /unlink 60000 (time 1749542016.83 total 72.03 last 645.48) total: 64768 /unlink in 79.02 seconds: 819.62 ops/second Waiting for MDT destroys to complete - open/close 4913 (time 1749542114.99 total 10.00 last 491.28) - open/close 10000 (time 1749542124.75 total 19.76 last 521.38) - open/close 15180 (time 1749542134.75 total 29.76 last 517.95) - open/close 20000 (time 1749542144.06 total 39.06 last 517.91) - open/close 25503 (time 1749542154.06 total 49.07 last 550.26) - open/close 30000 (time 1749542162.14 total 57.15 last 556.47) - open/close 35616 (time 1749542172.14 total 67.15 last 561.55) - open/close 40000 (time 1749542179.96 total 74.96 last 560.92) - open/close 45372 (time 1749542189.96 total 84.96 last 537.19) - open/close 50000 (time 1749542198.50 total 93.51 last 541.64) - open/close 55363 (time 1749542208.50 total 103.51 last 536.20) - open/close 60000 (time 1749542217.03 total 112.04 last 543.66) total: 64768 open/close in 120.82 seconds: 536.06 ops/second - /unlink 8793 (time 1749542236.66 total 10.00 last 879.22) - /unlink 17922 (time 1749542246.66 total 20.00 last 912.87) - /unlink 20000 (time 1749542249.21 total 22.55 last 816.70) - /unlink 28492 (time 1749542259.21 total 32.55 last 849.15) - /unlink 38071 (time 1749542269.21 total 42.55 last 957.83) - /unlink 47969 (time 1749542279.21 total 52.55 last 989.75) - /unlink 50000 (time 1749542281.27 total 54.61 last 986.82) - /unlink 59946 (time 1749542291.27 total 64.61 last 994.51) total: 64768 /unlink in 69.47 seconds: 932.35 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2