== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:02:08 (1721217728) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg307-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 6055 (time 1721217768.59 total 10.00 last 605.47) - open/close 10000 (time 1721217775.12 total 16.53 last 603.87) - open/close 16017 (time 1721217785.12 total 26.53 last 601.64) - open/close 20000 (time 1721217791.68 total 33.09 last 607.81) - open/close 25872 (time 1721217801.69 total 43.10 last 586.51) - open/close 30000 (time 1721217808.48 total 49.89 last 608.12) - open/close 36096 (time 1721217818.48 total 59.89 last 609.56) - open/close 40000 (time 1721217825.00 total 66.41 last 598.51) - open/close 46040 (time 1721217835.00 total 76.41 last 603.96) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 82.49 seconds: 602.44 ops/second - /unlink 10000 (time 1721217851.08 total 9.15 last 1093.11) - /unlink 20000 (time 1721217859.61 total 17.67 last 1173.13) - /unlink 30000 (time 1721217868.16 total 26.22 last 1169.31) - /unlink 40000 (time 1721217876.97 total 35.03 last 1135.23) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 43.59 seconds: 1140.10 ops/second - open/close 6016 (time 1721217896.35 total 10.00 last 601.51) - open/close 10000 (time 1721217903.36 total 17.02 last 567.75) - open/close 15804 (time 1721217913.36 total 27.02 last 580.33) - open/close 20000 (time 1721217920.69 total 34.34 last 573.09) - open/close 25355 (time 1721217930.69 total 44.34 last 535.44) - open/close 30000 (time 1721217938.93 total 52.58 last 563.70) - open/close 35424 (time 1721217948.93 total 62.58 last 542.39) - open/close 40000 (time 1721217956.98 total 70.63 last 568.44) - open/close 45994 (time 1721217966.98 total 80.63 last 599.39) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 86.82 seconds: 572.45 ops/second - /unlink 10000 (time 1721217983.04 total 9.04 last 1106.58) - /unlink 20000 (time 1721217992.12 total 18.12 last 1101.28) - /unlink 30000 (time 1721218000.97 total 26.97 last 1129.46) - /unlink 40000 (time 1721218009.78 total 35.78 last 1135.57) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 43.93 seconds: 1131.32 ops/second - open/close 6196 (time 1721218028.91 total 10.00 last 619.57) - open/close 10000 (time 1721218034.92 total 16.01 last 632.75) - open/close 16167 (time 1721218044.92 total 26.01 last 616.59) - open/close 20000 (time 1721218051.22 total 32.31 last 608.42) - open/close 25929 (time 1721218061.22 total 42.31 last 592.90) - open/close 30000 (time 1721218068.32 total 49.42 last 573.09) - open/close 36128 (time 1721218078.32 total 59.42 last 612.73) - open/close 40000 (time 1721218084.59 total 65.68 last 618.14) - open/close 46002 (time 1721218094.59 total 75.68 last 600.14) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 81.90 seconds: 606.82 ops/second - /unlink 10000 (time 1721218110.68 total 9.01 last 1109.38) - /unlink 20000 (time 1721218119.68 total 18.01 last 1111.21) - /unlink 30000 (time 1721218129.49 total 27.83 last 1019.00) - /unlink 40000 (time 1721218138.36 total 36.70 last 1127.21) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 45.02 seconds: 1103.96 ops/second - open/close 5536 (time 1721218157.54 total 10.00 last 553.56) - open/close 9131 (time 1721218167.54 total 20.00 last 359.48) - open/close 13654 (time 1721218177.54 total 30.00 last 452.23) - open/close 17501 (time 1721218187.54 total 40.00 last 384.70) - open/close 20000 (time 1721218192.33 total 44.80 last 521.48) - open/close 23861 (time 1721218202.33 total 54.80 last 386.09) - open/close 27315 (time 1721218212.34 total 64.80 last 345.15) - open/close 30000 (time 1721218217.38 total 69.84 last 532.54) - open/close 33156 (time 1721218227.38 total 79.84 last 315.60) - open/close 37633 (time 1721218237.39 total 89.85 last 447.57) - open/close 40000 (time 1721218244.11 total 96.58 last 351.82) - open/close 44123 (time 1721218254.12 total 106.58 last 412.22) - open/close 47813 (time 1721218264.12 total 116.58 last 368.99) open(/mnt/lustre/d106.conf-sanity/f-49295) error: No space left on device total: 49295 open/close in 121.57 seconds: 405.49 ops/second - /unlink 8402 (time 1721218280.40 total 10.00 last 840.18) - /unlink 18420 (time 1721218290.40 total 20.00 last 1001.74) - /unlink 28464 (time 1721218300.40 total 30.00 last 1004.39) - /unlink 38737 (time 1721218310.40 total 40.00 last 1027.29) unlink(/mnt/lustre/d106.conf-sanity/f-49295) error: No such file or directory total: 49295 /unlink in 49.99 seconds: 986.18 ops/second - open/close 3339 (time 1721218331.47 total 10.00 last 333.84) - open/close 9132 (time 1721218341.47 total 20.00 last 579.19) - open/close 15052 (time 1721218351.48 total 30.00 last 591.93) - open/close 20000 (time 1721218361.27 total 39.80 last 505.22) - open/close 24096 (time 1721218371.27 total 49.80 last 409.53) - open/close 27742 (time 1721218381.27 total 59.80 last 364.57) - open/close 30000 (time 1721218386.95 total 65.48 last 397.45) - open/close 34112 (time 1721218396.95 total 75.48 last 411.16) - open/close 38219 (time 1721218406.95 total 85.48 last 410.68) - open/close 40000 (time 1721218412.22 total 90.75 last 338.13) - open/close 43463 (time 1721218422.22 total 100.75 last 346.23) - open/close 46762 (time 1721218432.23 total 110.76 last 329.84) open(/mnt/lustre/d106.conf-sanity/f-49295) error: No space left on device total: 49295 open/close in 116.42 seconds: 423.42 ops/second - /unlink 8185 (time 1721218449.13 total 10.00 last 818.39) - /unlink 10000 (time 1721218451.33 total 12.20 last 824.01) - /unlink 20000 (time 1721218460.91 total 21.78 last 1044.10) - /unlink 27859 (time 1721218470.91 total 31.78 last 785.76) - /unlink 30000 (time 1721218474.59 total 35.47 last 581.32) - /unlink 37851 (time 1721218484.59 total 45.47 last 785.06) - /unlink 40000 (time 1721218487.10 total 47.98 last 855.68) unlink(/mnt/lustre/d106.conf-sanity/f-49295) error: No such file or directory total: 49295 /unlink in 56.39 seconds: 874.22 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server LNET ready to unload unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: LNET ready to unload modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2