== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:52 (1734259672) Stopping clients: oleg312-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg312-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 oleg312-server: oleg312-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing load_modules_local oleg312-server: Loading modules from /home/green/git/lustre-release/lustre oleg312-server: detected 4 online CPUs by sysfs oleg312-server: Force libcfs to create 2 CPU partitions oleg312-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg312-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg312-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg312-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2336 (time 1734259731.34 total 10.00 last 233.55) - open/close 4477 (time 1734259741.34 total 20.00 last 214.04) - open/close 6662 (time 1734259751.34 total 30.01 last 218.48) - open/close 8782 (time 1734259761.34 total 40.01 last 211.99) - open/close 10000 (time 1734259766.31 total 44.97 last 245.29) - open/close 12492 (time 1734259776.31 total 54.97 last 249.14) - open/close 15069 (time 1734259786.31 total 64.98 last 257.65) - open/close 17630 (time 1734259796.31 total 74.98 last 256.03) - open/close 19782 (time 1734259806.31 total 84.98 last 215.20) - open/close 22092 (time 1734259816.31 total 94.98 last 230.97) - open/close 24634 (time 1734259826.32 total 104.98 last 254.17) - open/close 27396 (time 1734259836.32 total 114.98 last 276.18) - open/close 30000 (time 1734259845.73 total 124.39 last 276.65) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 133.46 seconds: 243.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4993 (time 1734259865.88 total 10.00 last 499.28) - /unlink 9181 (time 1734259875.88 total 20.00 last 418.77) - /unlink 13293 (time 1734259885.88 total 30.00 last 411.15) - /unlink 17291 (time 1734259895.89 total 40.01 last 399.66) - /unlink 20000 (time 1734259903.04 total 47.16 last 378.73) - /unlink 24401 (time 1734259913.04 total 57.16 last 440.08) - /unlink 29241 (time 1734259923.04 total 67.16 last 483.95) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 74.07 seconds: 439.59 ops/second Waiting for MDT destroys to complete oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 - open/close 2548 (time 1734259948.47 total 10.00 last 254.73) - open/close 5091 (time 1734259958.48 total 20.00 last 254.27) - open/close 7554 (time 1734259968.48 total 30.01 last 246.22) - open/close 10000 (time 1734259977.86 total 39.38 last 260.86) - open/close 12640 (time 1734259987.86 total 49.38 last 264.00) - open/close 15481 (time 1734259997.86 total 59.38 last 284.10) - open/close 18820 (time 1734260007.86 total 69.39 last 333.84) - open/close 20000 (time 1734260011.43 total 72.96 last 330.48) - open/close 22943 (time 1734260021.43 total 82.96 last 294.27) - open/close 25848 (time 1734260031.43 total 92.96 last 290.44) - open/close 28765 (time 1734260041.43 total 102.96 last 291.64) - open/close 30000 (time 1734260046.21 total 107.74 last 258.46) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 115.79 seconds: 281.20 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7408 (time 1734260065.23 total 10.00 last 740.73) - /unlink 10000 (time 1734260069.09 total 13.86 last 671.52) - /unlink 15929 (time 1734260079.09 total 23.86 last 592.85) - /unlink 20000 (time 1734260085.86 total 30.63 last 601.46) - /unlink 25784 (time 1734260095.86 total 40.63 last 578.34) - /unlink 30000 (time 1734260103.11 total 47.88 last 581.82) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 52.00 seconds: 626.13 ops/second Waiting for MDT destroys to complete oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 - open/close 2933 (time 1734260133.68 total 10.00 last 293.28) - open/close 5897 (time 1734260143.68 total 20.00 last 296.37) - open/close 9195 (time 1734260153.68 total 30.00 last 329.75) - open/close 10000 (time 1734260155.98 total 32.30 last 349.92) - open/close 13645 (time 1734260165.99 total 42.31 last 364.45) - open/close 16902 (time 1734260175.99 total 52.31 last 325.65) - open/close 20000 (time 1734260185.30 total 61.62 last 332.60) - open/close 23455 (time 1734260195.30 total 71.62 last 345.43) - open/close 26448 (time 1734260205.30 total 81.62 last 299.27) - open/close 29530 (time 1734260215.31 total 91.63 last 308.13) - open/close 32503 (time 1734260225.31 total 101.63 last 297.28) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 101.81 seconds: 319.81 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5485 (time 1734260236.51 total 10.00 last 548.45) - /unlink 10000 (time 1734260245.00 total 18.49 last 532.11) - /unlink 15812 (time 1734260255.00 total 28.49 last 581.19) - /unlink 20000 (time 1734260261.53 total 35.01 last 641.68) - /unlink 25817 (time 1734260271.53 total 45.01 last 581.66) - /unlink 30000 (time 1734260279.53 total 53.02 last 522.76) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 57.74 seconds: 563.87 ops/second Waiting for MDT destroys to complete oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 - open/close 2609 (time 1734260300.94 total 10.00 last 260.84) - open/close 5406 (time 1734260310.94 total 20.00 last 279.66) - open/close 8130 (time 1734260320.94 total 30.01 last 272.33) - open/close 10000 (time 1734260326.96 total 36.03 last 310.54) - open/close 12921 (time 1734260336.97 total 46.03 last 292.06) - open/close 15912 (time 1734260346.97 total 56.03 last 299.07) - open/close 19048 (time 1734260356.97 total 66.03 last 313.59) - open/close 20000 (time 1734260360.28 total 69.34 last 287.56) - open/close 22795 (time 1734260370.28 total 79.34 last 279.44) - open/close 25485 (time 1734260380.28 total 89.34 last 269.00) - open/close 28111 (time 1734260390.28 total 99.35 last 262.52) - open/close 30000 (time 1734260397.80 total 106.87 last 251.22) - open/close 32433 (time 1734260407.80 total 116.87 last 243.25) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 117.49 seconds: 277.13 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5000 (time 1734260419.61 total 10.00 last 499.97) - /unlink 9990 (time 1734260429.61 total 20.00 last 498.94) - /unlink 14725 (time 1734260439.61 total 30.00 last 473.46) - /unlink 19369 (time 1734260449.62 total 40.00 last 464.34) - /unlink 24561 (time 1734260459.62 total 50.01 last 519.13) - /unlink 29856 (time 1734260469.62 total 60.01 last 529.48) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.89 seconds: 501.81 ops/second Waiting for MDT destroys to complete oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 - open/close 3068 (time 1734260494.09 total 10.00 last 306.73) - open/close 6166 (time 1734260504.09 total 20.00 last 309.77) - open/close 9154 (time 1734260514.09 total 30.00 last 298.76) - open/close 10000 (time 1734260516.83 total 32.75 last 308.48) - open/close 13018 (time 1734260526.83 total 42.75 last 301.79) - open/close 16378 (time 1734260536.84 total 52.75 last 335.96) - open/close 19096 (time 1734260546.84 total 62.75 last 271.80) - open/close 20000 (time 1734260550.41 total 66.33 last 252.72) - open/close 22880 (time 1734260560.42 total 76.33 last 287.87) - open/close 25963 (time 1734260570.42 total 86.33 last 308.29) - open/close 29174 (time 1734260580.42 total 96.33 last 321.05) - open/close 30000 (time 1734260582.62 total 98.54 last 374.66) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 105.83 seconds: 307.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5801 (time 1734260600.89 total 10.00 last 580.09) - /unlink 10000 (time 1734260608.64 total 17.75 last 541.65) - /unlink 16203 (time 1734260618.64 total 27.75 last 620.23) - /unlink 20000 (time 1734260624.64 total 33.75 last 633.16) - /unlink 25514 (time 1734260634.64 total 43.75 last 551.27) - /unlink 30000 (time 1734260642.31 total 51.42 last 584.73) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 55.28 seconds: 588.98 ops/second Waiting for MDT destroys to complete oleg312-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 oleg312-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server unloading modules via unload_modules_local on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2