== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:26:38 (1729088798) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2163 (time 1729088845.53 total 10.01 last 216.19) - open/close 4493 (time 1729088855.53 total 20.01 last 233.00) - open/close 6933 (time 1729088865.53 total 30.01 last 243.96) - open/close 9535 (time 1729088875.53 total 40.01 last 260.19) - open/close 10000 (time 1729088877.63 total 42.11 last 221.60) - open/close 12563 (time 1729088887.63 total 52.11 last 256.27) - open/close 15591 (time 1729088897.64 total 62.11 last 302.72) - open/close 18352 (time 1729088907.64 total 72.11 last 276.03) - open/close 20000 (time 1729088915.31 total 79.79 last 214.73) - open/close 22470 (time 1729088925.32 total 89.79 last 246.92) - open/close 25421 (time 1729088935.32 total 99.79 last 295.08) - open/close 28195 (time 1729088945.32 total 109.79 last 277.33) - open/close 30000 (time 1729088949.74 total 114.21 last 408.33) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 123.62 seconds: 263.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6132 (time 1729088970.13 total 10.00 last 613.12) - /unlink 10000 (time 1729088977.13 total 17.00 last 552.38) - /unlink 15955 (time 1729088987.13 total 27.00 last 595.46) - /unlink 20000 (time 1729088994.78 total 34.65 last 528.97) - /unlink 24268 (time 1729089004.78 total 44.65 last 426.74) - /unlink 30000 (time 1729089014.52 total 54.39 last 588.94) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 59.04 seconds: 551.53 ops/second Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 - open/close 2716 (time 1729089041.93 total 10.00 last 271.55) - open/close 5681 (time 1729089051.93 total 20.00 last 296.46) - open/close 8672 (time 1729089061.93 total 30.01 last 299.04) - open/close 10000 (time 1729089067.14 total 35.21 last 255.05) - open/close 12998 (time 1729089077.14 total 45.21 last 299.77) - open/close 15443 (time 1729089087.14 total 55.22 last 244.42) - open/close 18124 (time 1729089097.14 total 65.22 last 268.09) - open/close 20000 (time 1729089104.09 total 72.16 last 270.08) - open/close 23231 (time 1729089114.09 total 82.16 last 323.07) - open/close 26480 (time 1729089124.09 total 92.17 last 324.79) - open/close 29194 (time 1729089134.10 total 102.17 last 271.32) - open/close 30000 (time 1729089137.13 total 105.20 last 265.91) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 113.38 seconds: 287.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4697 (time 1729089156.34 total 10.00 last 469.64) - /unlink 8143 (time 1729089166.34 total 20.00 last 344.55) - /unlink 10000 (time 1729089171.67 total 25.34 last 348.17) - /unlink 13648 (time 1729089181.67 total 35.34 last 364.80) - /unlink 18744 (time 1729089191.67 total 45.34 last 509.48) - /unlink 20000 (time 1729089193.84 total 47.50 last 580.41) - /unlink 25897 (time 1729089203.84 total 57.50 last 589.62) - /unlink 30000 (time 1729089210.52 total 64.19 last 613.98) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 69.05 seconds: 471.53 ops/second Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 - open/close 3413 (time 1729089237.69 total 10.00 last 341.23) - open/close 6722 (time 1729089247.70 total 20.00 last 330.82) - open/close 10000 (time 1729089256.76 total 29.07 last 361.45) - open/close 13501 (time 1729089266.77 total 39.08 last 350.01) - open/close 17221 (time 1729089276.77 total 49.08 last 371.96) - open/close 20000 (time 1729089285.01 total 57.32 last 337.10) - open/close 23096 (time 1729089295.01 total 67.32 last 309.51) - open/close 26600 (time 1729089305.02 total 77.33 last 350.32) - open/close 29547 (time 1729089315.02 total 87.33 last 294.67) - open/close 30000 (time 1729089317.07 total 89.38 last 220.85) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 98.54 seconds: 330.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5623 (time 1729089337.24 total 10.00 last 562.29) - /unlink 10000 (time 1729089344.80 total 17.56 last 578.81) - /unlink 16212 (time 1729089354.80 total 27.56 last 621.09) - /unlink 20000 (time 1729089361.09 total 33.86 last 601.95) - /unlink 28468 (time 1729089371.09 total 43.86 last 846.69) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 49.59 seconds: 656.53 ops/second Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 - open/close 3165 (time 1729089407.25 total 10.00 last 316.47) - open/close 6165 (time 1729089417.25 total 20.00 last 299.93) - open/close 9250 (time 1729089427.25 total 30.00 last 308.47) - open/close 10000 (time 1729089429.71 total 32.47 last 304.49) - open/close 12549 (time 1729089439.71 total 42.47 last 254.90) - open/close 15325 (time 1729089449.71 total 52.47 last 277.56) - open/close 18202 (time 1729089459.71 total 62.47 last 287.67) - open/close 20000 (time 1729089465.63 total 68.38 last 304.06) - open/close 23695 (time 1729089475.63 total 78.38 last 369.48) - open/close 27756 (time 1729089485.63 total 88.38 last 406.07) - open/close 30000 (time 1729089490.84 total 93.60 last 430.55) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 99.90 seconds: 325.92 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7584 (time 1729089508.10 total 10.00 last 758.34) - /unlink 10000 (time 1729089511.14 total 13.05 last 793.02) - /unlink 18560 (time 1729089521.14 total 23.05 last 855.92) - /unlink 26682 (time 1729089531.15 total 33.05 last 812.17) - /unlink 30000 (time 1729089534.18 total 36.09 last 1092.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 38.45 seconds: 846.73 ops/second Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 - open/close 5318 (time 1729089577.86 total 10.00 last 531.79) - open/close 10000 (time 1729089586.06 total 18.20 last 571.13) - open/close 15803 (time 1729089596.06 total 28.20 last 580.29) - open/close 20000 (time 1729089604.39 total 36.53 last 503.74) - open/close 25385 (time 1729089614.39 total 46.53 last 538.45) - open/close 30000 (time 1729089623.08 total 55.22 last 531.14) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 60.56 seconds: 537.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9406 (time 1729089639.36 total 10.00 last 940.53) - /unlink 19220 (time 1729089649.36 total 20.00 last 981.33) - /unlink 29550 (time 1729089659.36 total 30.00 last 1032.86) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.82 seconds: 992.00 ops/second Waiting for MDT destroys to complete oleg254-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg254-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2