== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:15:14 (1715202914) Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg435-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5821 (time 1715202955.19 total 10.00 last 582.06) - open/close 10000 (time 1715202962.26 total 17.06 last 591.57) - open/close 15502 (time 1715202972.26 total 27.07 last 550.15) - open/close 20000 (time 1715202980.11 total 34.92 last 572.65) - open/close 25758 (time 1715202990.11 total 44.92 last 575.79) - open/close 30000 (time 1715202997.56 total 52.37 last 569.63) - open/close 35731 (time 1715203007.56 total 62.37 last 573.04) - open/close 40000 (time 1715203015.23 total 70.04 last 556.83) - open/close 45352 (time 1715203025.23 total 80.04 last 535.12) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 88.24 seconds: 563.19 ops/second - /unlink 10000 (time 1715203043.78 total 9.47 last 1055.46) - /unlink 15821 (time 1715203053.78 total 19.48 last 582.05) - /unlink 20000 (time 1715203061.25 total 26.94 last 559.57) - /unlink 28396 (time 1715203071.25 total 36.94 last 839.58) - /unlink 38106 (time 1715203081.25 total 46.94 last 970.92) - /unlink 40000 (time 1715203083.83 total 49.53 last 733.31) - /unlink 47356 (time 1715203093.83 total 59.53 last 735.58) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 62.68 seconds: 792.85 ops/second - open/close 3510 (time 1715203108.01 total 10.00 last 350.90) - open/close 8480 (time 1715203118.01 total 20.00 last 496.94) - open/close 10000 (time 1715203120.52 total 22.52 last 604.82) - open/close 14468 (time 1715203130.52 total 32.52 last 446.79) - open/close 18933 (time 1715203140.53 total 42.52 last 446.43) - open/close 20000 (time 1715203143.55 total 45.55 last 352.41) - open/close 23466 (time 1715203153.55 total 55.55 last 346.53) - open/close 28257 (time 1715203163.56 total 65.55 last 479.06) - open/close 30000 (time 1715203166.89 total 68.88 last 523.20) - open/close 34000 (time 1715203176.89 total 78.88 last 399.94) - open/close 38912 (time 1715203186.89 total 88.88 last 491.13) - open/close 40000 (time 1715203188.93 total 90.92 last 534.47) - open/close 44745 (time 1715203198.93 total 100.92 last 474.45) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 110.45 seconds: 449.96 ops/second - /unlink 10000 (time 1715203218.57 total 9.23 last 1083.02) - /unlink 18987 (time 1715203228.57 total 19.23 last 898.68) - /unlink 28722 (time 1715203238.58 total 29.23 last 973.43) - /unlink 40000 (time 1715203248.32 total 38.98 last 1156.85) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 47.68 seconds: 1042.23 ops/second - open/close 5552 (time 1715203267.81 total 10.00 last 555.16) - open/close 10000 (time 1715203277.39 total 19.58 last 464.28) - open/close 15910 (time 1715203287.39 total 29.58 last 590.97) - open/close 20000 (time 1715203293.86 total 36.04 last 632.95) - open/close 25729 (time 1715203303.86 total 46.04 last 572.85) - open/close 30000 (time 1715203310.77 total 52.96 last 617.37) - open/close 36029 (time 1715203320.78 total 62.96 last 602.82) - open/close 40000 (time 1715203327.20 total 69.38 last 618.51) - open/close 46320 (time 1715203337.20 total 79.39 last 631.94) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 84.62 seconds: 587.32 ops/second - /unlink 10000 (time 1715203351.88 total 8.64 last 1156.82) - /unlink 20000 (time 1715203361.16 total 17.93 last 1077.01) - /unlink 30000 (time 1715203371.05 total 27.81 last 1011.77) - /unlink 39266 (time 1715203381.05 total 37.81 last 926.59) - /unlink 48763 (time 1715203391.05 total 47.81 last 949.63) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 48.80 seconds: 1018.39 ops/second - open/close 5536 (time 1715203402.92 total 10.00 last 553.60) - open/close 10000 (time 1715203411.67 total 18.75 last 510.35) - open/close 15707 (time 1715203421.67 total 28.75 last 570.64) - open/close 20000 (time 1715203429.12 total 36.21 last 575.58) - open/close 25247 (time 1715203439.13 total 46.21 last 524.64) - open/close 30000 (time 1715203448.07 total 55.15 last 531.63) - open/close 34810 (time 1715203458.07 total 65.15 last 480.93) - open/close 40000 (time 1715203467.45 total 74.53 last 553.44) - open/close 45778 (time 1715203477.45 total 84.53 last 577.79) open(/mnt/lustre/d106.conf-sanity/f-47557) error: No space left on device total: 47557 open/close in 87.24 seconds: 545.14 ops/second - /unlink 10000 (time 1715203489.77 total 8.83 last 1133.03) - /unlink 20000 (time 1715203497.97 total 17.02 last 1219.71) - /unlink 30000 (time 1715203506.61 total 25.67 last 1157.26) - /unlink 40000 (time 1715203516.14 total 35.19 last 1049.95) unlink(/mnt/lustre/d106.conf-sanity/f-47557) error: No such file or directory total: 47557 /unlink in 42.72 seconds: 1113.26 ops/second - open/close 6229 (time 1715203534.44 total 10.00 last 622.81) - open/close 10000 (time 1715203540.48 total 16.04 last 624.86) - open/close 15862 (time 1715203550.48 total 26.04 last 586.11) - open/close 20000 (time 1715203557.50 total 33.06 last 589.63) - open/close 25809 (time 1715203567.50 total 43.06 last 580.84) - open/close 30000 (time 1715203574.16 total 49.72 last 628.87) - open/close 36054 (time 1715203584.16 total 59.72 last 605.36) - open/close 40000 (time 1715203591.19 total 66.75 last 561.76) - open/close 45739 (time 1715203601.19 total 76.75 last 573.85) open(/mnt/lustre/d106.conf-sanity/f-47557) error: No space left on device total: 47557 open/close in 80.47 seconds: 590.98 ops/second - /unlink 10000 (time 1715203615.30 total 9.47 last 1055.83) - /unlink 20000 (time 1715203623.58 total 17.75 last 1207.71) - /unlink 30000 (time 1715203631.50 total 25.67 last 1262.44) - /unlink 40000 (time 1715203639.40 total 33.57 last 1266.09) unlink(/mnt/lustre/d106.conf-sanity/f-47557) error: No such file or directory total: 47557 /unlink in 40.03 seconds: 1188.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server LNET ready to unload unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: LNET ready to unload modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2