== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:31:26 (1734258686) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 oleg437-server: oleg437-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg437-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg437-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3522 (time 1734258748.64 total 10.00 last 352.18) - open/close 6933 (time 1734258758.64 total 20.00 last 341.08) - open/close 10000 (time 1734258767.00 total 28.36 last 367.07) - open/close 13200 (time 1734258777.00 total 38.36 last 319.89) - open/close 16740 (time 1734258787.00 total 48.36 last 353.93) - open/close 19571 (time 1734258797.01 total 58.36 last 283.04) - open/close 22363 (time 1734258807.01 total 68.37 last 279.16) - open/close 25068 (time 1734258817.01 total 78.37 last 270.46) - open/close 28099 (time 1734258827.01 total 88.37 last 303.07) - open/close 30000 (time 1734258832.57 total 93.93 last 341.73) - open/close 32153 (time 1734258842.57 total 103.93 last 215.28) - open/close 34449 (time 1734258852.57 total 113.93 last 229.56) - open/close 36996 (time 1734258862.58 total 123.93 last 254.67) - open/close 40000 (time 1734258872.48 total 133.84 last 303.42) - open/close 43218 (time 1734258882.48 total 143.84 last 321.70) - open/close 46102 (time 1734258892.48 total 153.84 last 288.32) - open/close 48825 (time 1734258902.48 total 163.84 last 272.23) - open/close 50000 (time 1734258906.32 total 167.67 last 306.69) - open/close 52696 (time 1734258916.32 total 177.68 last 269.58) - open/close 55420 (time 1734258926.32 total 187.68 last 272.38) - open/close 59551 (time 1734258936.32 total 197.68 last 413.04) - open/close 62965 (time 1734258946.32 total 207.68 last 341.33) total: 64768 open/close in 213.34 seconds: 303.59 ops/second - /unlink 6070 (time 1734258962.89 total 10.00 last 606.93) - /unlink 10000 (time 1734258970.25 total 17.36 last 534.17) - /unlink 15334 (time 1734258980.25 total 27.36 last 533.31) - /unlink 20000 (time 1734258989.58 total 36.69 last 499.92) - /unlink 25028 (time 1734258999.58 total 46.69 last 502.73) - /unlink 30000 (time 1734259008.02 total 55.13 last 589.43) - /unlink 36152 (time 1734259018.02 total 65.13 last 615.19) - /unlink 40000 (time 1734259025.30 total 72.42 last 528.07) - /unlink 45295 (time 1734259035.31 total 82.42 last 529.43) - /unlink 50000 (time 1734259044.27 total 91.38 last 524.78) - /unlink 55070 (time 1734259054.27 total 101.39 last 506.91) - /unlink 60000 (time 1734259064.08 total 111.19 last 502.95) total: 64768 /unlink in 118.84 seconds: 545.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2774 (time 1734259099.51 total 10.00 last 277.39) - open/close 6247 (time 1734259109.51 total 20.00 last 347.29) - open/close 9674 (time 1734259119.51 total 30.00 last 342.66) - open/close 13562 (time 1734259129.51 total 40.00 last 388.75) - open/close 17419 (time 1734259139.51 total 50.00 last 385.68) - open/close 20000 (time 1734259146.34 total 56.83 last 378.24) - open/close 24464 (time 1734259156.34 total 66.83 last 446.35) - open/close 28728 (time 1734259166.34 total 76.83 last 426.32) - open/close 30000 (time 1734259168.98 total 79.47 last 481.93) - open/close 34475 (time 1734259178.98 total 89.47 last 447.41) - open/close 38886 (time 1734259188.98 total 99.47 last 441.05) - open/close 40000 (time 1734259191.25 total 101.74 last 490.71) - open/close 44844 (time 1734259201.25 total 111.74 last 484.38) - open/close 50000 (time 1734259211.09 total 121.58 last 524.32) - open/close 55022 (time 1734259221.09 total 131.58 last 502.18) - open/close 60000 (time 1734259230.34 total 140.83 last 538.12) total: 64768 open/close in 150.49 seconds: 430.37 ops/second - /unlink 7899 (time 1734259250.87 total 10.00 last 789.89) - /unlink 10000 (time 1734259253.51 total 12.63 last 798.43) - /unlink 18770 (time 1734259263.51 total 22.63 last 876.96) - /unlink 28326 (time 1734259273.51 total 32.63 last 955.56) - /unlink 36602 (time 1734259283.51 total 42.63 last 827.52) - /unlink 40000 (time 1734259287.28 total 46.41 last 900.49) - /unlink 48955 (time 1734259297.28 total 56.41 last 895.47) - /unlink 58687 (time 1734259307.28 total 66.41 last 972.99) total: 64768 /unlink in 72.57 seconds: 892.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4762 (time 1734259349.97 total 10.00 last 476.16) - open/close 9463 (time 1734259359.97 total 20.00 last 470.09) - open/close 14919 (time 1734259369.97 total 30.00 last 545.53) - open/close 19578 (time 1734259379.97 total 40.00 last 465.86) - open/close 23640 (time 1734259389.97 total 50.00 last 406.15) - open/close 27786 (time 1734259399.97 total 60.01 last 414.53) - open/close 30000 (time 1734259405.14 total 65.17 last 428.80) - open/close 34254 (time 1734259415.14 total 75.17 last 425.30) - open/close 38512 (time 1734259425.14 total 85.17 last 425.78) - open/close 40000 (time 1734259428.19 total 88.22 last 487.64) - open/close 44545 (time 1734259438.19 total 98.23 last 454.42) - open/close 48829 (time 1734259448.19 total 108.23 last 428.37) - open/close 50000 (time 1734259450.75 total 110.79 last 457.49) - open/close 54710 (time 1734259460.75 total 120.79 last 470.98) - open/close 58733 (time 1734259470.75 total 130.79 last 402.27) - open/close 60000 (time 1734259473.47 total 133.51 last 465.92) - open/close 64297 (time 1734259483.47 total 143.51 last 429.68) total: 64768 open/close in 144.61 seconds: 447.87 ops/second - /unlink 9612 (time 1734259495.38 total 10.00 last 961.18) - /unlink 19294 (time 1734259505.38 total 20.00 last 968.16) - /unlink 28726 (time 1734259515.39 total 30.00 last 943.13) - /unlink 38208 (time 1734259525.39 total 40.00 last 948.02) - /unlink 40000 (time 1734259527.43 total 42.05 last 877.64) - /unlink 49406 (time 1734259537.43 total 52.05 last 940.50) - /unlink 59266 (time 1734259547.43 total 62.05 last 985.96) total: 64768 /unlink in 67.78 seconds: 955.62 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5531 (time 1734259594.98 total 10.00 last 553.02) - open/close 10000 (time 1734259602.94 total 17.97 last 561.15) - open/close 15222 (time 1734259612.94 total 27.97 last 522.19) - open/close 19843 (time 1734259622.94 total 37.97 last 462.03) - open/close 24594 (time 1734259632.95 total 47.97 last 475.03) - open/close 29887 (time 1734259642.95 total 57.97 last 529.25) - open/close 34774 (time 1734259652.95 total 67.97 last 488.63) - open/close 39463 (time 1734259662.95 total 77.97 last 468.84) - open/close 44329 (time 1734259672.95 total 87.97 last 486.53) - open/close 48604 (time 1734259682.95 total 97.97 last 427.48) - open/close 50000 (time 1734259686.58 total 101.60 last 384.97) - open/close 54546 (time 1734259696.58 total 111.60 last 454.51) - open/close 59203 (time 1734259706.58 total 121.60 last 465.62) total: 64768 open/close in 131.35 seconds: 493.10 ops/second - /unlink 9350 (time 1734259727.13 total 10.00 last 934.94) - /unlink 17756 (time 1734259737.13 total 20.00 last 840.54) - /unlink 20000 (time 1734259739.57 total 22.45 last 917.13) - /unlink 29803 (time 1734259749.57 total 32.45 last 980.26) - /unlink 39159 (time 1734259759.57 total 42.45 last 935.59) - /unlink 47448 (time 1734259769.58 total 52.45 last 828.86) - /unlink 50000 (time 1734259772.47 total 55.34 last 882.75) - /unlink 58002 (time 1734259782.47 total 65.34 last 800.18) - /unlink 60000 (time 1734259784.88 total 67.76 last 826.33) total: 64768 /unlink in 73.34 seconds: 883.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5039 (time 1734259825.83 total 10.00 last 503.86) - open/close 10000 (time 1734259835.73 total 19.90 last 501.31) - open/close 15085 (time 1734259845.73 total 29.90 last 508.47) - open/close 20000 (time 1734259855.12 total 39.29 last 523.25) - open/close 25115 (time 1734259865.12 total 49.29 last 511.44) - open/close 30000 (time 1734259874.95 total 59.12 last 496.87) - open/close 35216 (time 1734259884.96 total 69.12 last 521.52) - open/close 40000 (time 1734259894.45 total 78.62 last 503.90) - open/close 45229 (time 1734259904.45 total 88.62 last 522.88) - open/close 50000 (time 1734259913.86 total 98.03 last 507.21) - open/close 54986 (time 1734259923.86 total 108.03 last 498.52) - open/close 59866 (time 1734259933.86 total 118.03 last 487.99) total: 64768 open/close in 127.62 seconds: 507.52 ops/second - /unlink 8836 (time 1734259954.30 total 10.00 last 883.57) - /unlink 17630 (time 1734259964.30 total 20.00 last 879.35) - /unlink 20000 (time 1734259966.89 total 22.59 last 914.84) - /unlink 28838 (time 1734259976.89 total 32.59 last 883.71) - /unlink 37514 (time 1734259986.89 total 42.59 last 867.57) - /unlink 40000 (time 1734259989.83 total 45.54 last 843.75) - /unlink 48615 (time 1734259999.83 total 55.54 last 861.47) - /unlink 57102 (time 1734260009.84 total 65.54 last 848.61) - /unlink 60000 (time 1734260013.17 total 68.87 last 869.61) total: 64768 /unlink in 75.35 seconds: 859.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2