== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:39:22 (1721381962) Stopping clients: oleg115-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg115-client.virtnet /mnt/lustre2 (opts:-f) oleg115-server: oleg115-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: detected 4 online CPUs by sysfs oleg115-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg115-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg115-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg115-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock oleg115-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5678 (time 1721382003.39 total 10.00 last 567.72) - open/close 10000 (time 1721382010.81 total 17.43 last 581.86) - open/close 15398 (time 1721382020.82 total 27.43 last 539.72) - open/close 20000 (time 1721382029.54 total 36.15 last 527.52) - open/close 25014 (time 1721382039.54 total 46.15 last 501.38) - open/close 30000 (time 1721382048.21 total 54.82 last 575.13) - open/close 35491 (time 1721382058.21 total 64.83 last 549.04) - open/close 40000 (time 1721382066.13 total 72.74 last 569.44) - open/close 45840 (time 1721382076.13 total 82.75 last 583.91) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 89.41 seconds: 555.87 ops/second - /unlink 10000 (time 1721382093.34 total 9.68 last 1033.14) - /unlink 20000 (time 1721382102.87 total 19.20 last 1050.20) - /unlink 30000 (time 1721382112.25 total 28.59 last 1065.41) - /unlink 40000 (time 1721382121.62 total 37.95 last 1067.65) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 47.20 seconds: 1052.84 ops/second - open/close 5480 (time 1721382141.77 total 10.00 last 547.99) - open/close 10000 (time 1721382149.47 total 17.70 last 587.02) - open/close 16031 (time 1721382159.47 total 27.70 last 603.04) - open/close 20000 (time 1721382167.07 total 35.30 last 522.43) - open/close 25914 (time 1721382177.07 total 45.30 last 591.32) - open/close 30000 (time 1721382183.98 total 52.21 last 591.35) - open/close 35798 (time 1721382193.98 total 62.21 last 579.77) - open/close 40000 (time 1721382201.26 total 69.49 last 577.12) - open/close 45808 (time 1721382211.26 total 79.49 last 580.74) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 86.45 seconds: 574.89 ops/second - /unlink 10000 (time 1721382228.42 total 9.32 last 1072.73) - /unlink 20000 (time 1721382237.77 total 18.67 last 1069.79) - /unlink 30000 (time 1721382247.00 total 27.90 last 1083.29) - /unlink 40000 (time 1721382256.27 total 37.17 last 1078.54) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 46.10 seconds: 1078.15 ops/second - open/close 5900 (time 1721382276.02 total 10.00 last 590.00) - open/close 10000 (time 1721382282.88 total 16.86 last 597.78) - open/close 16035 (time 1721382292.88 total 26.86 last 603.48) - open/close 20000 (time 1721382299.58 total 33.55 last 592.20) - open/close 25981 (time 1721382309.58 total 43.56 last 598.05) - open/close 30000 (time 1721382316.27 total 50.25 last 600.23) - open/close 35769 (time 1721382326.27 total 60.25 last 576.84) - open/close 40000 (time 1721382333.38 total 67.36 last 595.34) - open/close 45885 (time 1721382343.38 total 77.36 last 588.49) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 83.80 seconds: 593.03 ops/second - /unlink 10000 (time 1721382359.99 total 9.30 last 1075.33) - /unlink 20000 (time 1721382369.12 total 18.42 last 1095.83) - /unlink 30000 (time 1721382378.33 total 27.64 last 1085.25) - /unlink 40000 (time 1721382387.27 total 36.58 last 1118.69) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 45.33 seconds: 1096.48 ops/second - open/close 6044 (time 1721382406.86 total 10.00 last 604.34) - open/close 10000 (time 1721382413.52 total 16.65 last 594.60) - open/close 16008 (time 1721382423.52 total 26.65 last 600.79) - open/close 20000 (time 1721382430.20 total 33.34 last 596.93) - open/close 26062 (time 1721382440.20 total 43.34 last 606.18) - open/close 29698 (time 1721382450.20 total 53.34 last 363.59) - open/close 34947 (time 1721382460.21 total 63.34 last 524.87) - open/close 40000 (time 1721382469.81 total 72.95 last 526.15) - open/close 46091 (time 1721382479.81 total 82.95 last 609.00) open(/mnt/lustre/d106.conf-sanity/f-47555) error: No space left on device total: 47555 open/close in 85.32 seconds: 557.40 ops/second - /unlink 10000 (time 1721382491.94 total 8.90 last 1123.15) - /unlink 20000 (time 1721382500.33 total 17.29 last 1192.15) - /unlink 30000 (time 1721382508.75 total 25.72 last 1186.51) - /unlink 40000 (time 1721382517.34 total 34.31 last 1164.36) unlink(/mnt/lustre/d106.conf-sanity/f-47555) error: No such file or directory total: 47555 /unlink in 40.88 seconds: 1163.16 ops/second - open/close 3701 (time 1721382534.81 total 10.00 last 369.99) - open/close 8218 (time 1721382544.81 total 20.00 last 451.64) - open/close 10000 (time 1721382547.82 total 23.01 last 592.19) - open/close 14766 (time 1721382557.82 total 33.01 last 476.55) - open/close 18658 (time 1721382567.82 total 43.02 last 389.15) - open/close 20000 (time 1721382571.43 total 46.62 last 372.06) - open/close 25486 (time 1721382581.43 total 56.62 last 548.51) - open/close 29069 (time 1721382591.43 total 66.63 last 358.26) - open/close 30000 (time 1721382593.81 total 69.00 last 391.77) - open/close 34591 (time 1721382603.81 total 79.00 last 459.05) - open/close 39271 (time 1721382613.81 total 89.00 last 467.98) - open/close 43978 (time 1721382623.81 total 99.00 last 470.64) - open/close 47041 (time 1721382633.81 total 109.01 last 306.25) open(/mnt/lustre/d106.conf-sanity/f-47555) error: No space left on device total: 47555 open/close in 110.77 seconds: 429.31 ops/second - /unlink 9012 (time 1721382646.61 total 10.00 last 901.18) - /unlink 18934 (time 1721382656.61 total 20.00 last 992.15) - /unlink 30000 (time 1721382666.25 total 29.64 last 1147.59) - /unlink 40000 (time 1721382675.61 total 39.01 last 1068.05) unlink(/mnt/lustre/d106.conf-sanity/f-47555) error: No such file or directory total: 47555 /unlink in 45.61 seconds: 1042.59 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server LNET ready to unload unloading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: LNET ready to unload modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2