== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:51:13 (1736261473) Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f) oleg202-server: oleg202-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing load_modules_local oleg202-server: Loading modules from /home/green/git/lustre-release/lustre oleg202-server: detected 4 online CPUs by sysfs oleg202-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg202-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg202-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg202-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5040 (time 1736261518.85 total 10.00 last 503.98) - open/close 10000 (time 1736261527.13 total 18.29 last 598.69) - open/close 14534 (time 1736261537.13 total 28.29 last 453.34) - open/close 20000 (time 1736261546.79 total 37.95 last 565.92) - open/close 25749 (time 1736261556.79 total 47.95 last 574.83) - open/close 30000 (time 1736261564.68 total 55.83 last 539.27) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 60.76 seconds: 536.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736261579.78 total 9.24 last 1082.70) - /unlink 20000 (time 1736261589.44 total 18.90 last 1034.72) - /unlink 30000 (time 1736261599.25 total 28.71 last 1019.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.45 seconds: 1036.54 ops/second Waiting for MDT destroys to complete - open/close 5473 (time 1736261662.68 total 10.00 last 547.29) - open/close 10000 (time 1736261670.48 total 17.80 last 580.23) - open/close 16002 (time 1736261680.49 total 27.80 last 600.12) - open/close 20000 (time 1736261687.18 total 34.50 last 597.11) - open/close 25712 (time 1736261697.18 total 44.50 last 571.09) - open/close 30000 (time 1736261704.53 total 51.85 last 583.26) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 56.10 seconds: 581.00 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9959 (time 1736261719.62 total 10.00 last 995.85) - /unlink 20000 (time 1736261728.12 total 18.51 last 1180.37) - /unlink 30000 (time 1736261736.29 total 26.67 last 1224.64) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 28.59 seconds: 1139.85 ops/second Waiting for MDT destroys to complete - open/close 5736 (time 1736261783.00 total 10.00 last 573.54) - open/close 10000 (time 1736261790.32 total 17.32 last 582.39) - open/close 15430 (time 1736261800.32 total 27.32 last 542.98) - open/close 20000 (time 1736261808.21 total 35.22 last 578.74) - open/close 25333 (time 1736261818.21 total 45.22 last 533.28) - open/close 30000 (time 1736261827.43 total 54.43 last 506.53) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 60.16 seconds: 541.76 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8928 (time 1736261844.13 total 10.00 last 892.72) - /unlink 17827 (time 1736261854.13 total 20.00 last 889.90) - /unlink 20000 (time 1736261856.53 total 22.40 last 904.53) - /unlink 29604 (time 1736261866.53 total 32.40 last 960.33) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 35.63 seconds: 914.85 ops/second Waiting for MDT destroys to complete - open/close 5911 (time 1736261889.46 total 10.00 last 591.05) - open/close 10000 (time 1736261895.93 total 16.47 last 632.13) - open/close 16205 (time 1736261905.93 total 26.47 last 620.50) - open/close 20000 (time 1736261912.27 total 32.81 last 598.34) - open/close 26237 (time 1736261922.27 total 42.81 last 623.67) - open/close 30000 (time 1736261928.11 total 48.65 last 644.51) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 52.66 seconds: 618.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736261942.46 total 9.44 last 1059.29) - /unlink 20000 (time 1736261951.96 total 18.94 last 1052.71) - /unlink 30000 (time 1736261961.95 total 28.93 last 1000.89) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.73 seconds: 1027.26 ops/second Waiting for MDT destroys to complete - open/close 5838 (time 1736262023.53 total 10.00 last 583.74) - open/close 10000 (time 1736262030.54 total 17.01 last 593.82) - open/close 15926 (time 1736262040.54 total 27.01 last 592.58) - open/close 20000 (time 1736262047.45 total 33.92 last 589.51) - open/close 25461 (time 1736262057.45 total 43.92 last 546.07) - open/close 30000 (time 1736262065.20 total 51.67 last 585.78) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 55.98 seconds: 582.21 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736262079.76 total 9.40 last 1064.11) - /unlink 20000 (time 1736262089.32 total 18.95 last 1046.56) - /unlink 30000 (time 1736262099.01 total 28.65 last 1031.36) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.08 seconds: 1048.84 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server LNET ready to unload unloading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local oleg202-server: LNET ready to unload modules unloaded. pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2