== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:08:47 (1749121727) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 999 (time 1749121792.84 total 10.00 last 99.89) - open/close 1971 (time 1749121802.85 total 20.01 last 97.13) - open/close 2975 (time 1749121812.86 total 30.02 last 100.32) - open/close 3944 (time 1749121822.86 total 40.02 last 96.86) - open/close 4898 (time 1749121832.89 total 50.04 last 95.17) - open/close 5867 (time 1749121842.89 total 60.05 last 96.86) - open/close 6960 (time 1749121852.90 total 70.05 last 109.22) - open/close 8065 (time 1749121862.90 total 80.06 last 110.47) - open/close 9138 (time 1749121872.91 total 90.07 last 107.19) - open/close 10000 (time 1749121881.33 total 98.49 last 102.40) - open/close 11111 (time 1749121891.34 total 108.49 last 111.01) - open/close 12164 (time 1749121901.34 total 118.50 last 105.25) - open/close 13223 (time 1749121911.34 total 128.50 last 105.89) - open/close 14257 (time 1749121921.34 total 138.50 last 103.38) - open/close 15287 (time 1749121931.35 total 148.51 last 102.93) - open/close 16338 (time 1749121941.36 total 158.51 last 105.03) - open/close 17410 (time 1749121951.37 total 168.52 last 107.09) - open/close 18598 (time 1749121961.39 total 178.54 last 118.59) - open/close 19786 (time 1749121971.39 total 188.55 last 118.72) - open/close 20925 (time 1749121981.40 total 198.55 last 113.87) - open/close 22097 (time 1749121991.40 total 208.56 last 117.13) - open/close 23288 (time 1749122001.41 total 218.56 last 119.05) - open/close 24532 (time 1749122011.41 total 228.56 last 124.39) - open/close 25810 (time 1749122021.41 total 238.57 last 127.78) - open/close 27004 (time 1749122031.41 total 248.57 last 119.38) - open/close 28212 (time 1749122041.41 total 258.57 last 120.77) - open/close 29601 (time 1749122051.41 total 268.57 last 138.89) - open/close 30000 (time 1749122054.27 total 271.42 last 139.88) - open/close 31336 (time 1749122064.27 total 281.42 last 133.59) - open/close 32643 (time 1749122074.27 total 291.42 last 130.69) - open/close 34140 (time 1749122084.27 total 301.43 last 149.66) - open/close 35593 (time 1749122094.27 total 311.43 last 145.29) - open/close 37067 (time 1749122104.27 total 321.43 last 147.36) - open/close 38549 (time 1749122114.27 total 331.43 last 148.20) - open/close 39974 (time 1749122124.27 total 341.43 last 142.48) - open/close 41497 (time 1749122134.28 total 351.44 last 152.22) - open/close 43108 (time 1749122144.29 total 361.44 last 160.97) - open/close 44696 (time 1749122154.29 total 371.44 last 158.79) - open/close 46116 (time 1749122164.29 total 381.45 last 141.92) - open/close 47469 (time 1749122174.29 total 391.45 last 135.28) - open/close 48494 (time 1749122184.30 total 401.45 last 102.49) - open/close 49714 (time 1749122194.30 total 411.46 last 121.96) - open/close 50000 (time 1749122196.45 total 413.60 last 133.08) - open/close 51411 (time 1749122206.45 total 423.61 last 141.08) - open/close 52916 (time 1749122216.45 total 433.61 last 150.49) - open/close 54483 (time 1749122226.46 total 443.62 last 156.47) - open/close 56091 (time 1749122236.46 total 453.62 last 160.79) - open/close 57795 (time 1749122246.46 total 463.62 last 170.40) - open/close 59527 (time 1749122256.47 total 473.62 last 173.19) - open/close 60000 (time 1749122259.34 total 476.50 last 164.41) - open/close 61631 (time 1749122269.34 total 486.50 last 163.08) - open/close 63219 (time 1749122279.35 total 496.50 last 158.73) total: 64768 open/close in 505.32 seconds: 128.17 ops/second - /unlink 3058 (time 1749122299.48 total 10.00 last 305.71) - /unlink 5940 (time 1749122309.48 total 20.01 last 288.11) - /unlink 9199 (time 1749122319.49 total 30.01 last 325.86) - /unlink 10000 (time 1749122322.35 total 32.87 last 279.83) - /unlink 13005 (time 1749122332.35 total 42.87 last 300.44) - /unlink 16023 (time 1749122342.35 total 52.87 last 301.77) - /unlink 19450 (time 1749122352.36 total 62.88 last 342.56) - /unlink 23724 (time 1749122362.36 total 72.88 last 427.32) - /unlink 28923 (time 1749122372.36 total 82.88 last 519.85) - /unlink 30000 (time 1749122374.68 total 85.20 last 464.12) - /unlink 34204 (time 1749122384.68 total 95.20 last 420.39) - /unlink 38675 (time 1749122394.68 total 105.20 last 447.08) - /unlink 40000 (time 1749122397.64 total 108.16 last 447.41) - /unlink 44032 (time 1749122407.64 total 118.16 last 403.15) - /unlink 47756 (time 1749122417.64 total 128.16 last 372.37) - /unlink 50000 (time 1749122422.99 total 133.51 last 419.85) - /unlink 54775 (time 1749122432.99 total 143.51 last 477.45) - /unlink 60000 (time 1749122442.82 total 153.34 last 531.69) total: 64768 /unlink in 161.51 seconds: 401.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2593 (time 1749122481.31 total 10.01 last 259.07) - open/close 4634 (time 1749122491.32 total 20.01 last 204.01) - open/close 7461 (time 1749122501.32 total 30.02 last 282.64) - open/close 10000 (time 1749122508.55 total 37.24 last 351.20) - open/close 13360 (time 1749122518.55 total 47.25 last 335.89) - open/close 16376 (time 1749122528.55 total 57.25 last 301.54) - open/close 19961 (time 1749122538.56 total 67.25 last 358.47) - open/close 23368 (time 1749122548.56 total 77.25 last 340.68) - open/close 26470 (time 1749122558.56 total 87.25 last 310.17) - open/close 29695 (time 1749122568.56 total 97.25 last 322.46) - open/close 33065 (time 1749122578.56 total 107.26 last 336.94) - open/close 36049 (time 1749122588.56 total 117.26 last 298.35) - open/close 39259 (time 1749122598.56 total 127.26 last 320.96) - open/close 40000 (time 1749122600.63 total 129.33 last 358.59) - open/close 43932 (time 1749122610.63 total 139.33 last 393.16) - open/close 47761 (time 1749122620.63 total 149.33 last 382.86) - open/close 50000 (time 1749122625.83 total 154.52 last 431.05) - open/close 54237 (time 1749122635.83 total 164.52 last 423.57) - open/close 58248 (time 1749122645.83 total 174.53 last 401.05) - open/close 60000 (time 1749122650.05 total 178.75 last 414.80) - open/close 63626 (time 1749122660.06 total 188.75 last 362.45) total: 64768 open/close in 192.62 seconds: 336.24 ops/second - /unlink 6669 (time 1749122674.89 total 10.00 last 666.88) - /unlink 10000 (time 1749122679.27 total 14.38 last 760.96) - /unlink 17653 (time 1749122689.27 total 24.38 last 765.21) - /unlink 20000 (time 1749122691.96 total 27.08 last 870.11) - /unlink 28341 (time 1749122701.96 total 37.08 last 834.08) - /unlink 30000 (time 1749122704.05 total 39.16 last 795.15) - /unlink 35967 (time 1749122714.05 total 49.16 last 596.61) - /unlink 40000 (time 1749122719.09 total 54.21 last 799.78) - /unlink 47751 (time 1749122729.09 total 64.21 last 775.10) - /unlink 50000 (time 1749122731.95 total 67.06 last 787.47) - /unlink 56334 (time 1749122741.95 total 77.06 last 633.38) - /unlink 60000 (time 1749122747.18 total 82.29 last 700.79) total: 64768 /unlink in 88.84 seconds: 729.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3836 (time 1749122781.88 total 10.00 last 383.54) - open/close 7823 (time 1749122791.88 total 20.00 last 398.62) - open/close 10000 (time 1749122798.18 total 26.31 last 345.43) - open/close 13874 (time 1749122808.18 total 36.31 last 387.31) - open/close 16958 (time 1749122818.19 total 46.31 last 308.31) - open/close 19979 (time 1749122828.19 total 56.31 last 302.08) - open/close 22754 (time 1749122838.19 total 66.31 last 277.47) - open/close 25634 (time 1749122848.19 total 76.31 last 287.99) - open/close 28638 (time 1749122858.19 total 86.31 last 300.36) - open/close 30000 (time 1749122861.49 total 89.62 last 412.51) - open/close 34439 (time 1749122871.49 total 99.62 last 443.82) - open/close 38286 (time 1749122881.50 total 109.62 last 384.65) - open/close 40000 (time 1749122885.77 total 113.89 last 401.07) - open/close 44836 (time 1749122895.77 total 123.89 last 483.60) - open/close 49404 (time 1749122905.81 total 133.93 last 455.16) - open/close 53835 (time 1749122915.81 total 143.93 last 443.02) - open/close 58006 (time 1749122925.81 total 153.93 last 417.07) - open/close 60000 (time 1749122930.23 total 158.36 last 450.67) - open/close 64631 (time 1749122940.23 total 168.36 last 463.00) total: 64768 open/close in 168.76 seconds: 383.78 ops/second - /unlink 8148 (time 1749122951.51 total 10.00 last 814.74) - /unlink 10000 (time 1749122953.82 total 12.31 last 802.71) - /unlink 18372 (time 1749122963.82 total 22.31 last 837.17) - /unlink 26891 (time 1749122973.82 total 32.31 last 851.81) - /unlink 30000 (time 1749122977.80 total 36.29 last 781.79) - /unlink 37740 (time 1749122987.80 total 46.29 last 773.99) - /unlink 40000 (time 1749122990.21 total 48.70 last 938.21) - /unlink 48630 (time 1749123000.21 total 58.70 last 862.97) - /unlink 57102 (time 1749123010.21 total 68.70 last 847.20) - /unlink 60000 (time 1749123013.63 total 72.11 last 847.71) total: 64768 /unlink in 77.27 seconds: 838.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5268 (time 1749123052.15 total 10.00 last 526.72) - open/close 10000 (time 1749123061.34 total 19.19 last 515.02) - open/close 14691 (time 1749123071.34 total 29.19 last 469.07) - open/close 19349 (time 1749123081.34 total 39.19 last 465.73) - open/close 23465 (time 1749123091.34 total 49.19 last 411.57) - open/close 26837 (time 1749123101.34 total 59.19 last 337.18) - open/close 30000 (time 1749123109.22 total 67.07 last 401.51) - open/close 33808 (time 1749123119.22 total 77.07 last 380.68) - open/close 38517 (time 1749123129.23 total 87.08 last 470.85) - open/close 40000 (time 1749123131.95 total 89.80 last 543.44) - open/close 45036 (time 1749123141.95 total 99.80 last 503.59) - open/close 50000 (time 1749123151.38 total 109.23 last 526.90) - open/close 54859 (time 1749123161.38 total 119.23 last 485.82) - open/close 60000 (time 1749123170.76 total 128.61 last 547.86) total: 64768 open/close in 138.32 seconds: 468.26 ops/second - /unlink 9739 (time 1749123191.30 total 10.00 last 973.82) - /unlink 19860 (time 1749123201.30 total 20.00 last 1012.05) - /unlink 28144 (time 1749123211.30 total 30.00 last 828.40) - /unlink 30000 (time 1749123213.46 total 32.15 last 862.31) - /unlink 38376 (time 1749123223.46 total 42.15 last 837.51) - /unlink 46606 (time 1749123233.46 total 52.16 last 822.98) - /unlink 50000 (time 1749123237.56 total 56.26 last 827.32) - /unlink 58467 (time 1749123247.56 total 66.26 last 846.65) total: 64768 /unlink in 73.53 seconds: 880.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5071 (time 1749123292.63 total 10.00 last 507.07) - open/close 9968 (time 1749123302.63 total 20.00 last 489.68) - open/close 14997 (time 1749123312.63 total 30.00 last 502.83) - open/close 20000 (time 1749123322.35 total 39.72 last 514.79) - open/close 24983 (time 1749123332.35 total 49.72 last 498.25) - open/close 29958 (time 1749123342.35 total 59.72 last 497.46) - open/close 35022 (time 1749123352.35 total 69.72 last 506.31) - open/close 40000 (time 1749123362.17 total 79.54 last 506.91) - open/close 44952 (time 1749123372.17 total 89.55 last 495.15) - open/close 49921 (time 1749123382.17 total 99.55 last 496.89) - open/close 54633 (time 1749123392.20 total 109.57 last 470.05) - open/close 59524 (time 1749123402.20 total 119.57 last 489.09) - open/close 64437 (time 1749123412.23 total 129.61 last 489.56) total: 64768 open/close in 130.29 seconds: 497.09 ops/second - /unlink 8039 (time 1749123423.78 total 10.00 last 803.88) - /unlink 10000 (time 1749123426.11 total 12.33 last 840.04) - /unlink 18460 (time 1749123436.11 total 22.33 last 845.99) - /unlink 27170 (time 1749123446.11 total 32.34 last 870.91) - /unlink 30000 (time 1749123449.34 total 35.57 last 876.31) - /unlink 38707 (time 1749123459.34 total 45.57 last 870.68) - /unlink 47391 (time 1749123469.34 total 55.57 last 868.32) - /unlink 50000 (time 1749123472.33 total 58.56 last 872.48) - /unlink 58859 (time 1749123482.33 total 68.56 last 885.85) total: 64768 /unlink in 75.14 seconds: 862.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2