== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:38:44 (1721407124) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) oleg409-server: oleg409-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg409-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg409-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg409-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock oleg409-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5903 (time 1721407164.45 total 10.00 last 590.27) - open/close 10000 (time 1721407171.32 total 16.87 last 596.22) - open/close 15938 (time 1721407181.32 total 26.87 last 593.77) - open/close 20000 (time 1721407190.24 total 35.79 last 455.55) - open/close 24408 (time 1721407200.24 total 45.79 last 440.74) - open/close 29452 (time 1721407210.24 total 55.79 last 504.33) - open/close 33855 (time 1721407220.24 total 65.79 last 440.27) - open/close 38716 (time 1721407230.25 total 75.79 last 486.03) - open/close 40000 (time 1721407232.87 total 78.42 last 489.70) - open/close 45035 (time 1721407242.87 total 88.42 last 503.44) - open/close 49045 (time 1721407252.87 total 98.42 last 400.90) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 99.96 seconds: 497.18 ops/second - /unlink 8331 (time 1721407265.71 total 10.00 last 833.07) - /unlink 17378 (time 1721407275.71 total 20.00 last 904.61) - /unlink 20000 (time 1721407278.33 total 22.62 last 1001.35) - /unlink 30000 (time 1721407287.52 total 31.81 last 1087.63) - /unlink 40000 (time 1721407296.51 total 40.80 last 1112.82) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 49.60 seconds: 1002.03 ops/second - open/close 5942 (time 1721407316.12 total 10.00 last 594.18) - open/close 10000 (time 1721407322.87 total 16.74 last 601.94) - open/close 16040 (time 1721407332.87 total 26.74 last 603.95) - open/close 20000 (time 1721407339.39 total 33.26 last 607.38) - open/close 25996 (time 1721407349.39 total 43.26 last 599.54) - open/close 30000 (time 1721407356.10 total 49.98 last 596.32) - open/close 35722 (time 1721407366.10 total 59.98 last 572.19) - open/close 40000 (time 1721407373.33 total 67.20 last 592.30) - open/close 45881 (time 1721407383.33 total 77.20 last 588.10) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 83.77 seconds: 593.30 ops/second - /unlink 9643 (time 1721407400.73 total 10.00 last 964.28) - /unlink 17492 (time 1721407410.74 total 20.00 last 784.88) - /unlink 20000 (time 1721407413.03 total 22.29 last 1093.17) - /unlink 29886 (time 1721407423.03 total 32.30 last 988.56) - /unlink 36385 (time 1721407433.03 total 42.30 last 649.87) - /unlink 40000 (time 1721407438.75 total 48.02 last 632.05) - /unlink 46871 (time 1721407448.75 total 58.02 last 687.08) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 61.28 seconds: 811.03 ops/second - open/close 4092 (time 1721407463.21 total 10.00 last 409.18) - open/close 7469 (time 1721407473.21 total 20.00 last 337.65) - open/close 10000 (time 1721407478.16 total 24.95 last 511.52) - open/close 15714 (time 1721407488.16 total 34.95 last 571.29) - open/close 20000 (time 1721407496.97 total 43.76 last 486.50) - open/close 25783 (time 1721407506.97 total 53.76 last 578.20) - open/close 30000 (time 1721407515.50 total 62.30 last 494.16) - open/close 35774 (time 1721407525.51 total 72.30 last 577.31) - open/close 40000 (time 1721407534.45 total 81.24 last 472.44) - open/close 45909 (time 1721407544.45 total 91.24 last 590.84) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 97.75 seconds: 508.41 ops/second - /unlink 10000 (time 1721407560.81 total 9.00 last 1110.87) - /unlink 20000 (time 1721407569.68 total 17.87 last 1127.23) - /unlink 30000 (time 1721407578.64 total 26.82 last 1117.18) - /unlink 40000 (time 1721407587.51 total 35.70 last 1126.89) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 44.19 seconds: 1124.56 ops/second - open/close 5967 (time 1721407606.90 total 10.00 last 596.70) - open/close 10000 (time 1721407615.41 total 18.52 last 473.63) - open/close 16064 (time 1721407625.41 total 28.52 last 606.39) - open/close 20000 (time 1721407631.97 total 35.08 last 599.94) - open/close 26008 (time 1721407641.97 total 45.08 last 600.79) - open/close 30000 (time 1721407648.80 total 51.90 last 584.85) - open/close 36000 (time 1721407658.80 total 61.90 last 599.96) - open/close 40000 (time 1721407665.53 total 68.64 last 593.98) - open/close 45348 (time 1721407675.54 total 78.64 last 534.75) open(/mnt/lustre/d106.conf-sanity/f-48621) error: No space left on device total: 48621 open/close in 84.53 seconds: 575.17 ops/second - /unlink 10000 (time 1721407691.82 total 9.40 last 1063.78) - /unlink 19297 (time 1721407701.82 total 19.40 last 929.62) - /unlink 30000 (time 1721407711.42 total 28.99 last 1115.85) - /unlink 40000 (time 1721407720.16 total 37.74 last 1143.16) unlink(/mnt/lustre/d106.conf-sanity/f-48621) error: No such file or directory total: 48621 /unlink in 47.58 seconds: 1021.87 ops/second - open/close 3866 (time 1721407741.23 total 10.00 last 386.57) - open/close 9227 (time 1721407751.23 total 20.00 last 536.07) - open/close 14345 (time 1721407761.23 total 30.00 last 511.76) - open/close 18478 (time 1721407771.24 total 40.01 last 413.15) - open/close 20000 (time 1721407776.40 total 45.17 last 294.97) - open/close 25023 (time 1721407786.40 total 55.17 last 502.24) - open/close 30000 (time 1721407796.36 total 65.13 last 499.76) - open/close 35085 (time 1721407806.36 total 75.13 last 508.47) - open/close 39334 (time 1721407816.36 total 85.13 last 424.84) - open/close 40000 (time 1721407818.57 total 87.34 last 301.27) - open/close 44488 (time 1721407828.57 total 97.34 last 448.72) open(/mnt/lustre/d106.conf-sanity/f-48621) error: No space left on device total: 48621 open/close in 105.59 seconds: 460.45 ops/second - /unlink 7689 (time 1721407847.65 total 10.00 last 768.84) - /unlink 10000 (time 1721407850.17 total 12.52 last 917.31) - /unlink 18051 (time 1721407860.17 total 22.52 last 805.05) - /unlink 29114 (time 1721407870.17 total 32.52 last 1106.21) - /unlink 40000 (time 1721407879.86 total 42.21 last 1123.47) - /unlink 46209 (time 1721407889.86 total 52.21 last 620.85) unlink(/mnt/lustre/d106.conf-sanity/f-48621) error: No such file or directory total: 48621 /unlink in 55.41 seconds: 877.49 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server LNET ready to unload unloading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: LNET ready to unload modules unloaded. pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2