== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:22:56 (1773102176) Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) oleg346-server: oleg346-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg346-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg346-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1461 (time 1773102249.52 total 10.00 last 146.10) - open/close 2873 (time 1773102259.52 total 20.00 last 141.16) - open/close 4391 (time 1773102269.52 total 30.00 last 151.79) - open/close 6018 (time 1773102279.53 total 40.01 last 162.58) - open/close 7492 (time 1773102289.53 total 50.01 last 147.37) - open/close 8980 (time 1773102299.54 total 60.02 last 148.67) - open/close 10000 (time 1773102306.61 total 67.09 last 144.31) - open/close 11444 (time 1773102316.61 total 77.09 last 144.35) - open/close 13209 (time 1773102326.61 total 87.10 last 176.47) - open/close 14962 (time 1773102336.62 total 97.10 last 175.25) - open/close 16652 (time 1773102346.62 total 107.10 last 168.96) - open/close 18304 (time 1773102356.63 total 117.11 last 165.09) - open/close 20000 (time 1773102365.64 total 126.12 last 188.20) - open/close 21721 (time 1773102375.64 total 136.12 last 172.06) - open/close 23454 (time 1773102385.64 total 146.12 last 173.28) - open/close 25164 (time 1773102395.64 total 156.12 last 170.99) - open/close 26884 (time 1773102405.65 total 166.13 last 171.92) - open/close 28608 (time 1773102415.65 total 176.13 last 172.30) - open/close 30000 (time 1773102423.60 total 184.08 last 175.10) - open/close 31888 (time 1773102433.60 total 194.08 last 188.79) - open/close 33792 (time 1773102443.60 total 204.08 last 190.38) - open/close 35660 (time 1773102453.60 total 214.09 last 186.75) - open/close 37605 (time 1773102463.61 total 224.09 last 194.45) - open/close 39562 (time 1773102473.61 total 234.09 last 195.69) - open/close 41835 (time 1773102483.61 total 244.09 last 227.28) - open/close 44109 (time 1773102493.61 total 254.09 last 227.32) - open/close 46012 (time 1773102503.61 total 264.10 last 190.26) - open/close 47750 (time 1773102513.62 total 274.10 last 173.72) - open/close 49580 (time 1773102523.62 total 284.10 last 182.94) - open/close 50000 (time 1773102526.38 total 286.86 last 152.44) - open/close 51799 (time 1773102536.38 total 296.86 last 179.84) - open/close 53513 (time 1773102546.39 total 306.87 last 171.28) - open/close 55247 (time 1773102556.39 total 316.87 last 173.35) - open/close 57096 (time 1773102566.39 total 326.87 last 184.86) - open/close 58879 (time 1773102576.40 total 336.88 last 178.25) - open/close 60000 (time 1773102581.04 total 341.52 last 241.35) - open/close 62536 (time 1773102591.04 total 351.53 last 253.49) - open/close 64691 (time 1773102601.05 total 361.53 last 215.42) total: 64768 open/close in 361.94 seconds: 178.95 ops/second - /unlink 4183 (time 1773102613.17 total 10.00 last 418.25) - /unlink 8431 (time 1773102623.17 total 20.00 last 424.78) - /unlink 10000 (time 1773102627.10 total 23.93 last 399.42) - /unlink 14870 (time 1773102637.10 total 33.93 last 486.96) - /unlink 19232 (time 1773102647.10 total 43.94 last 435.93) - /unlink 23419 (time 1773102657.11 total 53.94 last 418.62) - /unlink 26713 (time 1773102667.11 total 63.94 last 329.40) - /unlink 30000 (time 1773102676.12 total 72.95 last 364.75) - /unlink 34886 (time 1773102686.12 total 82.95 last 488.54) - /unlink 40000 (time 1773102695.28 total 92.12 last 558.06) - /unlink 45942 (time 1773102705.28 total 102.12 last 594.13) - /unlink 50000 (time 1773102712.44 total 109.28 last 566.75) - /unlink 56218 (time 1773102722.45 total 119.28 last 621.71) - /unlink 60000 (time 1773102729.13 total 125.97 last 565.44) total: 64768 /unlink in 133.31 seconds: 485.83 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 - open/close 2571 (time 1773102755.65 total 10.00 last 257.03) - open/close 5026 (time 1773102765.65 total 20.01 last 245.44) - open/close 7675 (time 1773102775.65 total 30.01 last 264.85) - open/close 10000 (time 1773102783.47 total 37.82 last 297.43) - open/close 12626 (time 1773102793.47 total 47.83 last 262.53) - open/close 15124 (time 1773102803.48 total 57.83 last 249.63) - open/close 17970 (time 1773102813.48 total 67.84 last 284.55) - open/close 20000 (time 1773102819.75 total 74.10 last 323.82) - open/close 23244 (time 1773102829.75 total 84.11 last 324.34) - open/close 26254 (time 1773102839.75 total 94.11 last 300.91) - open/close 28908 (time 1773102849.75 total 104.11 last 265.38) - open/close 30000 (time 1773102854.13 total 108.48 last 249.74) - open/close 32830 (time 1773102864.13 total 118.49 last 282.93) - open/close 35636 (time 1773102874.13 total 128.49 last 280.59) - open/close 38503 (time 1773102884.13 total 138.49 last 286.69) - open/close 40000 (time 1773102889.75 total 144.11 last 266.34) - open/close 42798 (time 1773102899.75 total 154.11 last 279.79) - open/close 45522 (time 1773102909.75 total 164.11 last 272.39) - open/close 48327 (time 1773102919.75 total 174.11 last 280.47) - open/close 50000 (time 1773102925.08 total 179.44 last 314.01) - open/close 53120 (time 1773102935.08 total 189.44 last 311.92) - open/close 56355 (time 1773102945.08 total 199.44 last 323.41) - open/close 59425 (time 1773102955.09 total 209.44 last 306.92) - open/close 62592 (time 1773102965.09 total 219.45 last 316.63) total: 64768 open/close in 226.62 seconds: 285.80 ops/second - /unlink 5391 (time 1773102983.87 total 10.00 last 539.05) - /unlink 10000 (time 1773102993.43 total 19.55 last 482.51) - /unlink 15341 (time 1773103003.43 total 29.55 last 534.06) - /unlink 20000 (time 1773103010.37 total 36.50 last 670.90) - /unlink 25926 (time 1773103020.37 total 46.50 last 592.53) - /unlink 30000 (time 1773103027.65 total 53.78 last 559.78) - /unlink 36569 (time 1773103037.65 total 63.78 last 656.85) - /unlink 40000 (time 1773103042.07 total 68.20 last 776.00) - /unlink 48628 (time 1773103052.07 total 78.20 last 862.78) - /unlink 56733 (time 1773103062.07 total 88.20 last 810.47) - /unlink 60000 (time 1773103065.93 total 92.06 last 846.95) total: 64768 /unlink in 97.97 seconds: 661.07 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 - open/close 4093 (time 1773103093.03 total 10.00 last 409.24) - open/close 8085 (time 1773103103.03 total 20.00 last 399.12) - open/close 10000 (time 1773103107.33 total 24.30 last 445.87) - open/close 14945 (time 1773103117.33 total 34.30 last 494.46) - open/close 19944 (time 1773103127.33 total 44.30 last 499.89) - open/close 24832 (time 1773103137.33 total 54.30 last 488.80) - open/close 29410 (time 1773103147.33 total 64.30 last 457.73) - open/close 34429 (time 1773103157.33 total 74.30 last 501.88) - open/close 39581 (time 1773103167.33 total 84.30 last 515.15) - open/close 44648 (time 1773103177.34 total 94.30 last 506.62) - open/close 49607 (time 1773103187.34 total 104.30 last 495.88) - open/close 54511 (time 1773103197.34 total 114.31 last 490.36) - open/close 59818 (time 1773103207.34 total 124.31 last 530.61) - open/close 64482 (time 1773103217.34 total 134.31 last 466.34) total: 64768 open/close in 134.93 seconds: 480.02 ops/second - /unlink 7371 (time 1773103229.31 total 10.00 last 737.07) - /unlink 10000 (time 1773103233.15 total 13.83 last 685.90) - /unlink 17135 (time 1773103243.15 total 23.83 last 713.45) - /unlink 20000 (time 1773103247.14 total 27.82 last 718.36) - /unlink 26493 (time 1773103257.14 total 37.82 last 649.23) - /unlink 30000 (time 1773103262.93 total 43.61 last 605.81) - /unlink 36099 (time 1773103272.93 total 53.61 last 609.81) - /unlink 40000 (time 1773103278.94 total 59.63 last 648.64) - /unlink 45667 (time 1773103288.95 total 69.63 last 566.49) - /unlink 50000 (time 1773103295.16 total 75.85 last 696.82) - /unlink 57242 (time 1773103305.16 total 85.85 last 724.14) - /unlink 60000 (time 1773103308.85 total 89.54 last 747.93) total: 64768 /unlink in 96.87 seconds: 668.64 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 - open/close 4135 (time 1773103336.23 total 10.00 last 413.49) - open/close 9327 (time 1773103346.23 total 20.00 last 519.12) - open/close 14789 (time 1773103356.23 total 30.00 last 546.11) - open/close 20000 (time 1773103366.22 total 40.00 last 521.42) - open/close 24828 (time 1773103376.23 total 50.00 last 482.76) - open/close 30000 (time 1773103385.94 total 59.71 last 532.54) - open/close 35572 (time 1773103395.94 total 69.71 last 557.16) - open/close 40000 (time 1773103405.04 total 78.81 last 486.49) - open/close 45164 (time 1773103415.04 total 88.81 last 516.34) - open/close 49910 (time 1773103425.05 total 98.82 last 474.32) - open/close 54982 (time 1773103435.05 total 108.82 last 507.20) - open/close 59893 (time 1773103445.05 total 118.82 last 491.08) total: 64768 open/close in 128.38 seconds: 504.49 ops/second - /unlink 9258 (time 1773103465.88 total 10.00 last 925.79) - /unlink 18114 (time 1773103475.88 total 20.00 last 885.52) - /unlink 20000 (time 1773103478.06 total 22.18 last 866.71) - /unlink 27526 (time 1773103488.06 total 32.18 last 752.55) - /unlink 30000 (time 1773103491.02 total 35.13 last 836.98) - /unlink 38547 (time 1773103501.02 total 45.13 last 854.62) - /unlink 40000 (time 1773103503.03 total 47.15 last 721.06) - /unlink 48763 (time 1773103513.03 total 57.15 last 876.08) - /unlink 57272 (time 1773103523.03 total 67.15 last 850.87) - /unlink 60000 (time 1773103526.27 total 70.39 last 843.42) total: 64768 /unlink in 75.96 seconds: 852.64 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 - open/close 4509 (time 1773103554.24 total 10.00 last 450.84) - open/close 9147 (time 1773103564.24 total 20.00 last 463.72) - open/close 13775 (time 1773103574.24 total 30.00 last 462.76) - open/close 18151 (time 1773103584.24 total 40.01 last 437.54) - open/close 20000 (time 1773103588.78 total 44.54 last 407.49) - open/close 24295 (time 1773103598.78 total 54.54 last 429.42) - open/close 28483 (time 1773103608.78 total 64.55 last 418.72) - open/close 30000 (time 1773103612.30 total 68.06 last 431.70) - open/close 34398 (time 1773103622.30 total 78.06 last 439.76) - open/close 38988 (time 1773103632.30 total 88.06 last 458.92) - open/close 40000 (time 1773103634.58 total 90.34 last 443.95) - open/close 44512 (time 1773103644.58 total 100.34 last 451.12) - open/close 49045 (time 1773103654.58 total 110.35 last 453.23) - open/close 50000 (time 1773103656.67 total 112.43 last 457.80) - open/close 54412 (time 1773103666.67 total 122.43 last 441.15) - open/close 59019 (time 1773103676.67 total 132.43 last 460.69) - open/close 60000 (time 1773103679.09 total 134.86 last 404.93) - open/close 64745 (time 1773103689.10 total 144.86 last 474.41) total: 64768 open/close in 144.91 seconds: 446.96 ops/second - /unlink 7568 (time 1773103700.66 total 10.00 last 756.74) - /unlink 10000 (time 1773103703.75 total 13.09 last 786.99) - /unlink 18756 (time 1773103713.75 total 23.09 last 875.48) - /unlink 26835 (time 1773103723.75 total 33.09 last 807.90) - /unlink 30000 (time 1773103727.59 total 36.93 last 824.17) - /unlink 37851 (time 1773103737.59 total 46.93 last 785.08) - /unlink 40000 (time 1773103740.05 total 49.39 last 874.11) - /unlink 48105 (time 1773103750.05 total 59.39 last 810.40) - /unlink 50000 (time 1773103752.60 total 61.94 last 743.81) - /unlink 58537 (time 1773103762.60 total 71.94 last 853.70) total: 64768 /unlink in 79.06 seconds: 819.19 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2