== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:46:53 (1734259613) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 oleg156-server: oleg156-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: detected 4 online CPUs by sysfs oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg156-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg156-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg156-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1885 (time 1734259680.56 total 10.00 last 188.49) - open/close 3964 (time 1734259690.56 total 20.00 last 207.88) - open/close 5873 (time 1734259700.57 total 30.00 last 190.84) - open/close 7750 (time 1734259710.57 total 40.01 last 187.66) - open/close 9481 (time 1734259720.57 total 50.01 last 173.02) - open/close 10000 (time 1734259723.37 total 52.80 last 185.85) - open/close 11786 (time 1734259733.37 total 62.80 last 178.59) - open/close 13659 (time 1734259743.37 total 72.81 last 187.23) - open/close 15676 (time 1734259753.37 total 82.81 last 201.66) - open/close 17581 (time 1734259763.38 total 92.81 last 190.42) - open/close 19409 (time 1734259773.38 total 102.82 last 182.78) - open/close 20000 (time 1734259776.07 total 105.51 last 219.17) - open/close 22153 (time 1734259786.08 total 115.51 last 215.28) - open/close 24461 (time 1734259796.08 total 125.52 last 230.73) - open/close 26464 (time 1734259806.08 total 135.52 last 200.30) - open/close 28660 (time 1734259816.08 total 145.52 last 219.58) - open/close 30000 (time 1734259822.45 total 151.89 last 210.32) - open/close 31911 (time 1734259832.45 total 161.89 last 191.01) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 165.43 seconds: 196.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3236 (time 1734259847.39 total 10.00 last 323.54) - /unlink 6887 (time 1734259857.39 total 20.00 last 365.09) - /unlink 10000 (time 1734259866.57 total 29.18 last 339.16) - /unlink 13282 (time 1734259876.57 total 39.18 last 328.09) - /unlink 16737 (time 1734259886.58 total 49.19 last 345.41) - /unlink 20000 (time 1734259895.38 total 57.99 last 370.78) - /unlink 23626 (time 1734259905.38 total 67.99 last 362.59) - /unlink 27728 (time 1734259915.38 total 77.99 last 410.19) - /unlink 30000 (time 1734259920.44 total 83.05 last 448.92) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 89.15 seconds: 365.24 ops/second Waiting for MDT destroys to complete oleg156-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg156-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 - open/close 2299 (time 1734259946.85 total 10.00 last 229.87) - open/close 4624 (time 1734259956.86 total 20.00 last 232.46) - open/close 7038 (time 1734259966.86 total 30.00 last 241.40) - open/close 9321 (time 1734259976.86 total 40.01 last 228.24) - open/close 10000 (time 1734259979.71 total 42.86 last 238.11) - open/close 12271 (time 1734259989.71 total 52.86 last 227.07) - open/close 14639 (time 1734259999.71 total 62.86 last 236.75) - open/close 17030 (time 1734260009.72 total 72.86 last 239.02) - open/close 19346 (time 1734260019.72 total 82.87 last 231.48) - open/close 20000 (time 1734260022.47 total 85.62 last 237.88) - open/close 22336 (time 1734260032.47 total 95.62 last 233.51) - open/close 24596 (time 1734260042.47 total 105.62 last 226.00) - open/close 26908 (time 1734260052.48 total 115.62 last 231.17) - open/close 29067 (time 1734260062.48 total 125.63 last 215.83) - open/close 30000 (time 1734260067.28 total 130.43 last 194.28) - open/close 31844 (time 1734260077.28 total 140.43 last 184.39) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 144.12 seconds: 225.93 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3840 (time 1734260092.27 total 10.00 last 383.95) - /unlink 7495 (time 1734260102.28 total 20.01 last 365.34) - /unlink 10000 (time 1734260108.79 total 26.52 last 384.79) - /unlink 13969 (time 1734260118.79 total 36.52 last 396.84) - /unlink 18205 (time 1734260128.79 total 46.52 last 423.51) - /unlink 20000 (time 1734260132.87 total 50.60 last 440.15) - /unlink 24530 (time 1734260142.87 total 60.60 last 452.90) - /unlink 28989 (time 1734260152.87 total 70.60 last 445.89) - /unlink 30000 (time 1734260154.91 total 72.63 last 497.45) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 77.31 seconds: 421.15 ops/second Waiting for MDT destroys to complete oleg156-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg156-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 - open/close 2795 (time 1734260179.57 total 10.00 last 279.42) - open/close 5872 (time 1734260189.57 total 20.01 last 307.63) - open/close 8928 (time 1734260199.57 total 30.01 last 305.53) - open/close 10000 (time 1734260203.35 total 33.79 last 283.72) - open/close 13495 (time 1734260213.35 total 43.79 last 349.43) - open/close 16821 (time 1734260223.35 total 53.79 last 332.58) - open/close 20000 (time 1734260232.02 total 62.46 last 366.65) - open/close 23314 (time 1734260242.02 total 72.46 last 331.36) - open/close 26264 (time 1734260252.02 total 82.46 last 295.00) - open/close 29355 (time 1734260262.03 total 92.46 last 309.05) - open/close 32546 (time 1734260272.03 total 102.47 last 318.98) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 102.50 seconds: 317.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6399 (time 1734260283.03 total 10.00 last 639.87) - /unlink 10000 (time 1734260288.60 total 15.57 last 646.57) - /unlink 15999 (time 1734260298.60 total 25.57 last 599.89) - /unlink 20000 (time 1734260306.38 total 33.35 last 514.19) - /unlink 24867 (time 1734260316.38 total 43.35 last 486.65) - /unlink 29589 (time 1734260326.38 total 53.35 last 472.10) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 58.76 seconds: 554.15 ops/second Waiting for MDT destroys to complete oleg156-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg156-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 - open/close 2754 (time 1734260350.47 total 10.00 last 275.34) - open/close 5893 (time 1734260360.47 total 20.00 last 313.86) - open/close 9331 (time 1734260370.47 total 30.00 last 343.77) - open/close 12544 (time 1734260380.47 total 40.01 last 321.25) - open/close 15710 (time 1734260390.47 total 50.01 last 316.58) - open/close 18933 (time 1734260400.47 total 60.01 last 322.25) - open/close 20000 (time 1734260403.28 total 62.82 last 379.81) - open/close 23107 (time 1734260413.28 total 72.82 last 310.67) - open/close 25758 (time 1734260423.29 total 82.82 last 265.00) - open/close 28645 (time 1734260433.29 total 92.83 last 288.60) - open/close 30000 (time 1734260438.39 total 97.92 last 265.81) - open/close 32496 (time 1734260448.39 total 107.93 last 249.47) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 108.22 seconds: 300.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4704 (time 1734260459.97 total 10.00 last 470.36) - /unlink 9322 (time 1734260469.97 total 20.00 last 461.76) - /unlink 13403 (time 1734260479.98 total 30.00 last 408.06) - /unlink 17846 (time 1734260489.98 total 40.00 last 444.25) - /unlink 20000 (time 1734260494.91 total 44.93 last 436.91) - /unlink 24547 (time 1734260504.91 total 54.94 last 454.63) - /unlink 29665 (time 1734260514.91 total 64.94 last 511.66) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 69.84 seconds: 466.22 ops/second Waiting for MDT destroys to complete oleg156-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg156-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 - open/close 3090 (time 1734260541.96 total 10.00 last 308.93) - open/close 6147 (time 1734260551.97 total 20.00 last 305.66) - open/close 9061 (time 1734260561.97 total 30.01 last 291.33) - open/close 10000 (time 1734260564.74 total 32.78 last 339.05) - open/close 13494 (time 1734260574.74 total 42.78 last 349.39) - open/close 17116 (time 1734260584.74 total 52.78 last 362.16) - open/close 20000 (time 1734260593.57 total 61.61 last 326.63) - open/close 23071 (time 1734260603.57 total 71.61 last 307.06) - open/close 27056 (time 1734260613.57 total 81.61 last 398.41) - open/close 30000 (time 1734260621.38 total 89.41 last 377.28) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 96.76 seconds: 336.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9199 (time 1734260639.68 total 10.00 last 919.84) - /unlink 15162 (time 1734260649.68 total 20.00 last 596.21) - /unlink 20000 (time 1734260658.22 total 28.53 last 567.10) - /unlink 25448 (time 1734260668.22 total 38.53 last 544.74) - /unlink 30000 (time 1734260676.12 total 46.44 last 575.72) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 50.83 seconds: 640.55 ops/second Waiting for MDT destroys to complete oleg156-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg156-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg156-server unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2