== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:56:40 (1717153000) Stopping clients: oleg148-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg148-client.virtnet /mnt/lustre2 (opts:-f) oleg148-server: oleg148-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg148-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg148-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg148-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock oleg148-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4889 (time 1717153041.95 total 10.00 last 488.80) - open/close 9596 (time 1717153051.96 total 20.00 last 470.64) - open/close 14582 (time 1717153061.97 total 30.01 last 498.13) - open/close 19241 (time 1717153071.97 total 40.01 last 465.90) - open/close 24989 (time 1717153081.97 total 50.01 last 574.78) - open/close 30000 (time 1717153090.25 total 58.30 last 604.72) - open/close 35675 (time 1717153100.25 total 68.30 last 567.44) - open/close 40000 (time 1717153107.42 total 75.47 last 603.31) - open/close 45124 (time 1717153117.42 total 85.47 last 512.31) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 93.29 seconds: 532.71 ops/second - /unlink 9808 (time 1717153136.09 total 10.00 last 980.75) - /unlink 19454 (time 1717153146.09 total 20.00 last 964.57) - /unlink 28868 (time 1717153156.09 total 30.00 last 941.28) - /unlink 37973 (time 1717153166.09 total 40.00 last 910.48) - /unlink 40000 (time 1717153168.36 total 42.27 last 895.77) - /unlink 49128 (time 1717153178.36 total 52.27 last 912.78) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 52.99 seconds: 937.86 ops/second - open/close 5197 (time 1717153189.96 total 10.00 last 519.66) - open/close 10000 (time 1717153199.85 total 19.89 last 485.53) - open/close 14982 (time 1717153209.85 total 29.89 last 498.15) - open/close 19758 (time 1717153219.85 total 39.89 last 477.59) - open/close 24982 (time 1717153229.85 total 49.89 last 522.39) - open/close 30000 (time 1717153238.38 total 58.42 last 588.30) - open/close 35727 (time 1717153248.38 total 68.43 last 572.62) - open/close 40000 (time 1717153255.37 total 75.41 last 611.51) - open/close 45826 (time 1717153265.37 total 85.41 last 582.53) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 92.30 seconds: 538.42 ops/second - /unlink 10000 (time 1717153282.69 total 9.60 last 1042.16) - /unlink 20000 (time 1717153292.19 total 19.10 last 1052.48) - /unlink 30000 (time 1717153301.59 total 28.50 last 1063.48) - /unlink 40000 (time 1717153310.84 total 37.75 last 1080.93) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 47.11 seconds: 1054.82 ops/second - open/close 5656 (time 1717153331.09 total 10.00 last 565.52) - open/close 10000 (time 1717153338.97 total 17.88 last 551.29) - open/close 15509 (time 1717153348.97 total 27.88 last 550.85) - open/close 20000 (time 1717153356.79 total 35.70 last 574.57) - open/close 25589 (time 1717153366.79 total 45.70 last 558.87) - open/close 30000 (time 1717153374.55 total 53.46 last 568.07) - open/close 35561 (time 1717153384.55 total 63.46 last 556.10) - open/close 40000 (time 1717153392.34 total 71.25 last 569.73) - open/close 45544 (time 1717153402.35 total 81.26 last 553.97) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 89.00 seconds: 558.41 ops/second - /unlink 9973 (time 1717153420.95 total 10.00 last 997.24) - /unlink 20000 (time 1717153430.60 total 19.65 last 1038.91) - /unlink 29249 (time 1717153440.60 total 29.65 last 924.83) - /unlink 39999 (time 1717153450.60 total 39.65 last 1074.93) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 48.48 seconds: 1025.21 ops/second - open/close 5300 (time 1717153470.29 total 10.00 last 529.97) - open/close 10000 (time 1717153478.54 total 18.25 last 569.64) - open/close 15697 (time 1717153488.54 total 28.25 last 569.66) - open/close 20000 (time 1717153496.12 total 35.84 last 567.19) - open/close 25531 (time 1717153506.13 total 45.84 last 553.04) - open/close 30000 (time 1717153514.82 total 54.54 last 513.94) - open/close 35254 (time 1717153524.82 total 64.54 last 525.37) - open/close 39825 (time 1717153534.82 total 74.54 last 457.05) - open/close 44871 (time 1717153544.82 total 84.54 last 504.56) open(/mnt/lustre/d106.conf-sanity/f-49013) error: No space left on device total: 49013 open/close in 92.31 seconds: 530.98 ops/second - /unlink 10000 (time 1717153563.34 total 9.72 last 1028.61) - /unlink 20000 (time 1717153572.89 total 19.28 last 1046.35) - /unlink 30000 (time 1717153582.46 total 28.85 last 1044.86) - /unlink 40000 (time 1717153592.29 total 38.68 last 1017.75) unlink(/mnt/lustre/d106.conf-sanity/f-49013) error: No such file or directory total: 49013 /unlink in 46.83 seconds: 1046.60 ops/second - open/close 5856 (time 1717153611.30 total 10.00 last 585.57) - open/close 10000 (time 1717153619.24 total 17.94 last 521.76) - open/close 15858 (time 1717153629.24 total 27.94 last 585.70) - open/close 20000 (time 1717153636.68 total 35.38 last 557.03) - open/close 25727 (time 1717153646.68 total 45.38 last 572.60) - open/close 30000 (time 1717153654.30 total 53.00 last 560.93) - open/close 35632 (time 1717153664.30 total 63.00 last 563.18) - open/close 40000 (time 1717153672.18 total 70.89 last 553.74) - open/close 45518 (time 1717153682.18 total 80.89 last 551.79) open(/mnt/lustre/d106.conf-sanity/f-49013) error: No space left on device total: 49013 open/close in 87.30 seconds: 561.42 ops/second - /unlink 10000 (time 1717153699.37 total 9.88 last 1012.14) - /unlink 20000 (time 1717153708.98 total 19.49 last 1040.25) - /unlink 30000 (time 1717153718.57 total 29.08 last 1042.92) - /unlink 40000 (time 1717153727.81 total 38.33 last 1081.77) unlink(/mnt/lustre/d106.conf-sanity/f-49013) error: No such file or directory total: 49013 /unlink in 46.66 seconds: 1050.40 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server LNET ready to unload unloading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: LNET ready to unload modules unloaded. pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2