== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:38 (1734258578) Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 oleg316-server: oleg316-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg316-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg316-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1751 (time 1734258637.81 total 10.00 last 175.08) - open/close 3252 (time 1734258647.82 total 20.00 last 150.05) - open/close 4772 (time 1734258657.82 total 30.01 last 151.92) - open/close 6348 (time 1734258667.83 total 40.01 last 157.53) - open/close 7763 (time 1734258677.83 total 50.02 last 141.42) - open/close 9235 (time 1734258687.83 total 60.02 last 147.20) - open/close 10000 (time 1734258692.75 total 64.93 last 155.71) - open/close 11492 (time 1734258702.75 total 74.94 last 149.11) - open/close 12952 (time 1734258712.75 total 84.94 last 146.00) - open/close 14532 (time 1734258722.76 total 94.94 last 157.92) - open/close 16176 (time 1734258732.76 total 104.95 last 164.35) - open/close 17878 (time 1734258742.77 total 114.95 last 170.09) - open/close 19610 (time 1734258752.77 total 124.96 last 173.14) - open/close 20000 (time 1734258755.09 total 127.28 last 167.77) - open/close 21586 (time 1734258765.09 total 137.28 last 158.60) - open/close 23443 (time 1734258775.09 total 147.28 last 185.69) - open/close 24993 (time 1734258785.10 total 157.29 last 154.93) - open/close 26505 (time 1734258795.10 total 167.29 last 151.12) - open/close 28009 (time 1734258805.11 total 177.30 last 150.36) - open/close 29696 (time 1734258815.11 total 187.30 last 168.59) - open/close 31374 (time 1734258825.12 total 197.30 last 167.78) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 203.98 seconds: 159.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3370 (time 1734258843.14 total 10.00 last 336.95) - /unlink 6766 (time 1734258853.14 total 20.00 last 339.59) - /unlink 10000 (time 1734258862.38 total 29.24 last 349.96) - /unlink 13412 (time 1734258872.39 total 39.25 last 341.05) - /unlink 16220 (time 1734258882.39 total 49.25 last 280.79) - /unlink 19055 (time 1734258892.39 total 59.25 last 283.47) - /unlink 20000 (time 1734258895.81 total 62.67 last 276.18) - /unlink 22698 (time 1734258905.81 total 72.67 last 269.78) - /unlink 25379 (time 1734258915.81 total 82.67 last 268.04) - /unlink 28057 (time 1734258925.81 total 92.67 last 267.78) - /unlink 30000 (time 1734258932.87 total 99.73 last 275.36) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 108.80 seconds: 299.86 ops/second Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 - open/close 1607 (time 1734258962.39 total 10.00 last 160.66) - open/close 3175 (time 1734258972.39 total 20.01 last 156.75) - open/close 4754 (time 1734258982.40 total 30.01 last 157.85) - open/close 6536 (time 1734258992.40 total 40.01 last 178.16) - open/close 8316 (time 1734259002.40 total 50.01 last 177.99) - open/close 10000 (time 1734259010.60 total 58.22 last 205.25) - open/close 11732 (time 1734259020.61 total 68.22 last 173.09) - open/close 13326 (time 1734259030.61 total 78.23 last 159.37) - open/close 14896 (time 1734259040.61 total 88.23 last 156.97) - open/close 16443 (time 1734259050.62 total 98.23 last 154.64) - open/close 18064 (time 1734259060.62 total 108.23 last 162.05) - open/close 19578 (time 1734259070.62 total 118.23 last 151.39) - open/close 20000 (time 1734259073.65 total 121.27 last 139.17) - open/close 21473 (time 1734259083.65 total 131.27 last 147.29) - open/close 22980 (time 1734259093.66 total 141.27 last 150.64) - open/close 24392 (time 1734259103.66 total 151.27 last 141.18) - open/close 25927 (time 1734259113.67 total 161.28 last 153.42) - open/close 27756 (time 1734259123.67 total 171.28 last 182.83) - open/close 29278 (time 1734259133.68 total 181.29 last 152.11) - open/close 30000 (time 1734259138.47 total 186.08 last 150.60) - open/close 31484 (time 1734259148.47 total 196.09 last 148.33) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 203.31 seconds: 160.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3279 (time 1734259167.13 total 10.02 last 327.40) - /unlink 6521 (time 1734259177.13 total 20.02 last 324.16) - /unlink 9519 (time 1734259187.13 total 30.02 last 299.75) - /unlink 12411 (time 1734259197.14 total 40.02 last 289.11) - /unlink 15433 (time 1734259207.14 total 50.02 last 302.18) - /unlink 18270 (time 1734259217.14 total 60.02 last 283.65) - /unlink 20000 (time 1734259222.86 total 65.75 last 302.20) - /unlink 23127 (time 1734259232.87 total 75.75 last 312.62) - /unlink 26607 (time 1734259242.87 total 85.75 last 347.93) - /unlink 30000 (time 1734259250.51 total 93.40 last 443.70) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 99.46 seconds: 328.04 ops/second Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 - open/close 1682 (time 1734259277.21 total 10.00 last 168.20) - open/close 3650 (time 1734259287.21 total 20.00 last 196.77) - open/close 5377 (time 1734259297.22 total 30.01 last 172.60) - open/close 7157 (time 1734259307.22 total 40.01 last 177.93) - open/close 8862 (time 1734259317.23 total 50.02 last 170.42) - open/close 10000 (time 1734259323.56 total 56.35 last 179.74) - open/close 12131 (time 1734259333.56 total 66.35 last 213.03) - open/close 14101 (time 1734259343.57 total 76.36 last 196.88) - open/close 15729 (time 1734259353.57 total 86.36 last 162.76) - open/close 17352 (time 1734259363.58 total 96.36 last 162.23) - open/close 19052 (time 1734259373.58 total 106.37 last 169.94) - open/close 20000 (time 1734259379.14 total 111.93 last 170.50) - open/close 21642 (time 1734259389.15 total 121.93 last 164.09) - open/close 23405 (time 1734259399.15 total 131.94 last 176.24) - open/close 25078 (time 1734259409.15 total 141.94 last 167.24) - open/close 26707 (time 1734259419.15 total 151.94 last 162.87) - open/close 28372 (time 1734259429.16 total 161.94 last 166.47) - open/close 30000 (time 1734259439.12 total 171.91 last 163.42) - open/close 31689 (time 1734259449.12 total 181.91 last 168.84) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 187.83 seconds: 173.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3202 (time 1734259466.57 total 10.00 last 320.18) - /unlink 6431 (time 1734259476.57 total 20.00 last 322.88) - /unlink 10000 (time 1734259486.32 total 29.75 last 366.03) - /unlink 13428 (time 1734259496.32 total 39.75 last 342.75) - /unlink 16444 (time 1734259506.32 total 49.76 last 301.53) - /unlink 19956 (time 1734259516.33 total 59.76 last 351.17) - /unlink 23669 (time 1734259526.33 total 69.76 last 371.26) - /unlink 27991 (time 1734259536.33 total 79.76 last 432.09) - /unlink 30000 (time 1734259542.02 total 85.45 last 353.02) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 93.07 seconds: 350.54 ops/second Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 - open/close 1797 (time 1734259571.66 total 10.00 last 179.62) - open/close 3690 (time 1734259581.66 total 20.01 last 189.27) - open/close 5772 (time 1734259591.67 total 30.01 last 208.16) - open/close 7965 (time 1734259601.67 total 40.01 last 219.24) - open/close 10000 (time 1734259609.77 total 48.11 last 251.22) - open/close 12363 (time 1734259619.77 total 58.11 last 236.28) - open/close 14633 (time 1734259629.78 total 68.12 last 226.88) - open/close 16869 (time 1734259639.78 total 78.12 last 223.59) - open/close 19036 (time 1734259649.78 total 88.12 last 216.65) - open/close 20000 (time 1734259653.77 total 92.11 last 241.60) - open/close 22084 (time 1734259663.77 total 102.11 last 208.35) - open/close 24287 (time 1734259673.77 total 112.11 last 220.27) - open/close 26474 (time 1734259683.78 total 122.12 last 218.60) - open/close 28695 (time 1734259693.78 total 132.12 last 221.99) - open/close 30000 (time 1734259699.20 total 137.54 last 240.79) - open/close 32464 (time 1734259709.20 total 147.54 last 246.36) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 148.32 seconds: 219.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4019 (time 1734259721.10 total 10.00 last 401.85) - /unlink 7971 (time 1734259731.11 total 20.00 last 395.15) - /unlink 10000 (time 1734259736.83 total 25.73 last 354.24) - /unlink 13948 (time 1734259746.84 total 35.73 last 394.72) - /unlink 17433 (time 1734259756.84 total 45.73 last 348.43) - /unlink 20000 (time 1734259763.22 total 52.12 last 401.92) - /unlink 24238 (time 1734259773.23 total 62.12 last 423.69) - /unlink 28792 (time 1734259783.23 total 72.13 last 455.30) - /unlink 30000 (time 1734259785.85 total 74.75 last 460.37) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 80.36 seconds: 406.02 ops/second Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 - open/close 1853 (time 1734259807.57 total 10.00 last 185.29) - open/close 3929 (time 1734259817.58 total 20.00 last 207.58) - open/close 6335 (time 1734259827.58 total 30.00 last 240.60) - open/close 8776 (time 1734259837.58 total 40.00 last 244.04) - open/close 10000 (time 1734259842.42 total 44.85 last 252.83) - open/close 12553 (time 1734259852.42 total 54.85 last 255.29) - open/close 14941 (time 1734259862.42 total 64.85 last 238.74) - open/close 17069 (time 1734259872.43 total 74.85 last 212.73) - open/close 19124 (time 1734259882.43 total 84.85 last 205.43) - open/close 20000 (time 1734259887.05 total 89.48 last 189.45) - open/close 21907 (time 1734259897.06 total 99.49 last 190.58) - open/close 23821 (time 1734259907.06 total 109.49 last 191.38) - open/close 25958 (time 1734259917.06 total 119.49 last 213.67) - open/close 28267 (time 1734259927.06 total 129.49 last 230.89) - open/close 30000 (time 1734259934.79 total 137.22 last 224.19) - open/close 32057 (time 1734259944.79 total 147.22 last 205.68) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 150.05 seconds: 217.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4082 (time 1734259958.89 total 10.00 last 408.11) - /unlink 8005 (time 1734259968.89 total 20.00 last 392.28) - /unlink 10000 (time 1734259973.49 total 24.61 last 433.39) - /unlink 14293 (time 1734259983.49 total 34.61 last 429.17) - /unlink 18943 (time 1734259993.49 total 44.61 last 464.97) - /unlink 20000 (time 1734259995.56 total 46.67 last 511.91) - /unlink 25874 (time 1734260005.56 total 56.67 last 587.39) - /unlink 30000 (time 1734260012.76 total 63.88 last 572.97) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 68.96 seconds: 473.11 ops/second Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2