== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:26:02 (1734258362) Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 oleg433-server: oleg433-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg433-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3928 (time 1734258409.79 total 10.00 last 392.75) - open/close 7293 (time 1734258419.80 total 20.00 last 336.43) - open/close 10000 (time 1734258428.78 total 28.99 last 301.30) - open/close 13234 (time 1734258438.78 total 38.99 last 323.39) - open/close 15619 (time 1734258448.78 total 48.99 last 238.42) - open/close 17797 (time 1734258458.79 total 58.99 last 217.74) - open/close 20000 (time 1734258467.49 total 67.70 last 253.01) - open/close 23143 (time 1734258477.50 total 77.71 last 314.18) - open/close 25986 (time 1734258487.50 total 87.71 last 284.24) - open/close 28739 (time 1734258497.50 total 97.71 last 275.25) - open/close 30000 (time 1734258502.11 total 102.32 last 273.43) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 111.57 seconds: 292.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5273 (time 1734258522.50 total 10.00 last 527.24) - /unlink 10000 (time 1734258531.62 total 19.12 last 518.34) - /unlink 15912 (time 1734258541.62 total 29.12 last 591.08) - /unlink 20000 (time 1734258549.48 total 36.98 last 520.08) - /unlink 25062 (time 1734258559.48 total 46.98 last 506.15) - /unlink 30000 (time 1734258568.18 total 55.69 last 567.40) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 58.92 seconds: 553.70 ops/second Waiting for MDT destroys to complete oleg433-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg433-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 - open/close 3272 (time 1734258590.82 total 10.00 last 327.14) - open/close 6038 (time 1734258600.82 total 20.00 last 276.55) - open/close 8587 (time 1734258610.82 total 30.00 last 254.87) - open/close 10000 (time 1734258616.53 total 35.71 last 247.88) - open/close 12409 (time 1734258626.53 total 45.71 last 240.87) - open/close 15625 (time 1734258636.53 total 55.71 last 321.59) - open/close 18428 (time 1734258646.53 total 65.71 last 280.26) - open/close 20000 (time 1734258652.57 total 71.75 last 260.03) - open/close 22968 (time 1734258662.58 total 81.76 last 296.63) - open/close 26347 (time 1734258672.58 total 91.76 last 337.87) - open/close 29613 (time 1734258682.58 total 101.76 last 326.48) - open/close 32227 (time 1734258692.58 total 111.76 last 261.37) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 113.05 seconds: 288.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5337 (time 1734258704.88 total 10.00 last 533.49) - /unlink 9637 (time 1734258714.88 total 20.00 last 429.95) - /unlink 14823 (time 1734258724.88 total 30.01 last 518.57) - /unlink 20000 (time 1734258733.79 total 38.92 last 580.95) - /unlink 26812 (time 1734258743.80 total 48.92 last 681.16) - /unlink 30000 (time 1734258747.95 total 53.08 last 766.43) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 57.14 seconds: 571.00 ops/second Waiting for MDT destroys to complete oleg433-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg433-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 - open/close 3842 (time 1734258772.63 total 10.00 last 384.14) - open/close 6979 (time 1734258782.63 total 20.00 last 313.68) - open/close 10000 (time 1734258791.23 total 28.61 last 351.11) - open/close 12753 (time 1734258801.23 total 38.61 last 275.28) - open/close 15820 (time 1734258811.23 total 48.61 last 306.61) - open/close 18509 (time 1734258821.24 total 58.61 last 268.85) - open/close 20000 (time 1734258826.09 total 63.46 last 307.41) - open/close 23367 (time 1734258836.09 total 73.46 last 336.68) - open/close 25525 (time 1734258846.09 total 83.47 last 215.72) - open/close 28011 (time 1734258856.09 total 93.47 last 248.50) - open/close 30000 (time 1734258863.47 total 100.85 last 269.55) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 109.13 seconds: 298.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6950 (time 1734258882.73 total 10.00 last 694.90) - /unlink 10000 (time 1734258887.92 total 15.19 last 587.95) - /unlink 16583 (time 1734258897.92 total 25.19 last 658.25) - /unlink 20000 (time 1734258903.92 total 31.19 last 569.51) - /unlink 26868 (time 1734258913.92 total 41.19 last 686.78) - /unlink 30000 (time 1734258919.16 total 46.43 last 598.23) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 50.91 seconds: 640.83 ops/second Waiting for MDT destroys to complete oleg433-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg433-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 - open/close 3767 (time 1734258964.74 total 10.00 last 376.65) - open/close 7304 (time 1734258974.74 total 20.00 last 353.66) - open/close 10000 (time 1734258982.52 total 27.78 last 346.70) - open/close 13078 (time 1734258992.52 total 37.79 last 307.58) - open/close 16351 (time 1734259002.52 total 47.79 last 327.26) - open/close 20000 (time 1734259012.43 total 57.70 last 368.25) - open/close 23327 (time 1734259022.43 total 67.70 last 332.69) - open/close 27077 (time 1734259032.43 total 77.70 last 374.97) - open/close 30000 (time 1734259041.63 total 86.89 last 317.98) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 95.15 seconds: 342.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6278 (time 1734259060.90 total 10.00 last 627.78) - /unlink 10000 (time 1734259066.22 total 15.32 last 700.27) - /unlink 17699 (time 1734259076.22 total 25.32 last 769.87) - /unlink 20000 (time 1734259078.64 total 27.74 last 949.87) - /unlink 27847 (time 1734259088.64 total 37.74 last 784.60) - /unlink 30000 (time 1734259091.72 total 40.81 last 700.22) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 45.15 seconds: 722.67 ops/second Waiting for MDT destroys to complete oleg433-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg433-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 - open/close 4284 (time 1734259126.59 total 10.00 last 428.38) - open/close 9123 (time 1734259136.59 total 20.00 last 483.86) - open/close 13614 (time 1734259146.59 total 30.00 last 449.05) - open/close 18597 (time 1734259156.59 total 40.00 last 498.28) - open/close 20000 (time 1734259159.76 total 43.17 last 442.45) - open/close 24917 (time 1734259169.76 total 53.18 last 491.60) - open/close 29816 (time 1734259179.77 total 63.18 last 489.79) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 69.02 seconds: 472.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 10000 (time 1734259196.15 total 9.67 last 1034.48) - /unlink 20000 (time 1734259204.71 total 18.22 last 1168.61) - /unlink 30000 (time 1734259213.44 total 26.95 last 1145.34) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 29.40 seconds: 1109.56 ops/second Waiting for MDT destroys to complete oleg433-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg433-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2