== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:01:02 (1740484862) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg123-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3384 (time 1740484901.89 total 10.00 last 338.39) - open/close 6136 (time 1740484911.89 total 20.00 last 275.20) - open/close 8752 (time 1740484921.89 total 30.00 last 261.53) - open/close 10000 (time 1740484926.69 total 34.80 last 260.04) - open/close 12318 (time 1740484936.69 total 44.80 last 231.76) - open/close 15207 (time 1740484946.70 total 54.81 last 288.85) - open/close 18491 (time 1740484956.70 total 64.81 last 328.38) - open/close 20000 (time 1740484961.12 total 69.23 last 340.93) - open/close 22719 (time 1740484971.12 total 79.23 last 271.89) - open/close 26051 (time 1740484981.13 total 89.24 last 333.13) - open/close 29195 (time 1740484991.13 total 99.24 last 314.31) - open/close 30000 (time 1740484993.37 total 101.48 last 359.13) - open/close 33100 (time 1740485003.37 total 111.48 last 309.97) - open/close 36346 (time 1740485013.37 total 121.48 last 324.49) - open/close 38600 (time 1740485023.38 total 131.49 last 225.38) - open/close 40000 (time 1740485028.93 total 137.04 last 252.10) - open/close 42493 (time 1740485038.93 total 147.04 last 249.21) - open/close 44393 (time 1740485048.93 total 157.04 last 189.95) - open/close 46508 (time 1740485058.94 total 167.05 last 211.46) - open/close 48312 (time 1740485068.94 total 177.05 last 180.33) - open/close 50000 (time 1740485077.90 total 186.01 last 188.34) - open/close 51958 (time 1740485087.91 total 196.02 last 195.73) - open/close 53689 (time 1740485097.91 total 206.02 last 173.06) - open/close 56133 (time 1740485107.91 total 216.02 last 244.36) - open/close 58627 (time 1740485117.91 total 226.02 last 249.39) - open/close 60000 (time 1740485123.57 total 231.68 last 242.48) - open/close 62775 (time 1740485133.57 total 241.68 last 277.46) total: 64768 open/close in 246.96 seconds: 262.26 ops/second - /unlink 5871 (time 1740485149.88 total 10.00 last 587.03) - /unlink 10000 (time 1740485156.65 total 16.77 last 610.09) - /unlink 15050 (time 1740485166.66 total 26.77 last 504.85) - /unlink 20000 (time 1740485175.88 total 36.00 last 536.46) - /unlink 27249 (time 1740485185.88 total 46.00 last 724.89) - /unlink 30000 (time 1740485189.54 total 49.65 last 752.72) - /unlink 36054 (time 1740485199.54 total 59.66 last 605.24) - /unlink 40000 (time 1740485206.72 total 66.83 last 549.87) - /unlink 44734 (time 1740485216.72 total 76.83 last 473.35) - /unlink 49601 (time 1740485226.72 total 86.83 last 486.67) - /unlink 55136 (time 1740485236.72 total 96.83 last 553.48) - /unlink 60000 (time 1740485244.10 total 104.22 last 659.05) total: 64768 /unlink in 111.05 seconds: 583.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3361 (time 1740485279.31 total 10.00 last 336.07) - open/close 6645 (time 1740485289.31 total 20.00 last 328.33) - open/close 9994 (time 1740485299.31 total 30.00 last 334.88) - open/close 13755 (time 1740485309.32 total 40.00 last 376.06) - open/close 17940 (time 1740485319.32 total 50.01 last 418.47) - open/close 20000 (time 1740485324.65 total 55.34 last 386.33) - open/close 23510 (time 1740485334.65 total 65.34 last 350.94) - open/close 26541 (time 1740485344.65 total 75.34 last 303.04) - open/close 29953 (time 1740485354.65 total 85.34 last 341.11) - open/close 33172 (time 1740485364.66 total 95.34 last 321.88) - open/close 36889 (time 1740485374.66 total 105.35 last 371.64) - open/close 40000 (time 1740485382.10 total 112.79 last 417.83) - open/close 43865 (time 1740485392.10 total 122.79 last 386.44) - open/close 47669 (time 1740485402.11 total 132.79 last 380.33) - open/close 50000 (time 1740485408.37 total 139.06 last 371.96) - open/close 53655 (time 1740485418.37 total 149.06 last 365.45) - open/close 57828 (time 1740485428.38 total 159.07 last 417.21) - open/close 60000 (time 1740485433.93 total 164.62 last 390.81) - open/close 64225 (time 1740485443.93 total 174.62 last 422.48) total: 64768 open/close in 175.87 seconds: 368.27 ops/second - /unlink 6818 (time 1740485456.16 total 10.00 last 681.79) - /unlink 10000 (time 1740485460.36 total 14.20 last 756.77) - /unlink 17561 (time 1740485470.36 total 24.21 last 756.08) - /unlink 20000 (time 1740485473.35 total 27.20 last 815.68) - /unlink 27579 (time 1740485483.35 total 37.20 last 757.86) - /unlink 30000 (time 1740485486.71 total 40.55 last 721.23) - /unlink 38081 (time 1740485496.71 total 50.55 last 808.09) - /unlink 40000 (time 1740485499.17 total 53.01 last 779.91) - /unlink 48532 (time 1740485509.17 total 63.01 last 853.19) - /unlink 57048 (time 1740485519.17 total 73.01 last 851.59) - /unlink 60000 (time 1740485522.60 total 76.44 last 862.51) total: 64768 /unlink in 81.90 seconds: 790.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5056 (time 1740485560.27 total 10.00 last 505.60) - open/close 10000 (time 1740485569.96 total 19.70 last 509.91) - open/close 15228 (time 1740485579.96 total 29.70 last 522.71) - open/close 20000 (time 1740485589.34 total 39.07 last 509.07) - open/close 24904 (time 1740485599.34 total 49.07 last 490.36) - open/close 29831 (time 1740485609.34 total 59.07 last 492.70) - open/close 35061 (time 1740485619.34 total 69.07 last 522.91) - open/close 40000 (time 1740485629.08 total 78.82 last 507.04) - open/close 44774 (time 1740485639.08 total 88.82 last 477.39) - open/close 49591 (time 1740485649.08 total 98.82 last 481.66) - open/close 54038 (time 1740485659.08 total 108.82 last 444.69) - open/close 58443 (time 1740485669.08 total 118.82 last 440.45) - open/close 60000 (time 1740485672.31 total 122.04 last 482.44) total: 64768 open/close in 131.55 seconds: 492.33 ops/second - /unlink 8688 (time 1740485692.72 total 10.00 last 868.68) - /unlink 17457 (time 1740485702.72 total 20.00 last 876.88) - /unlink 20000 (time 1740485705.70 total 22.98 last 852.49) - /unlink 28325 (time 1740485715.70 total 32.98 last 832.49) - /unlink 36977 (time 1740485725.70 total 42.98 last 865.20) - /unlink 40000 (time 1740485729.14 total 46.43 last 878.58) - /unlink 48536 (time 1740485739.14 total 56.43 last 853.56) - /unlink 57268 (time 1740485749.14 total 66.43 last 873.12) - /unlink 60000 (time 1740485752.43 total 69.71 last 831.71) total: 64768 /unlink in 75.14 seconds: 861.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5084 (time 1740485791.02 total 10.00 last 508.40) - open/close 9926 (time 1740485801.02 total 20.00 last 484.16) - open/close 14934 (time 1740485811.02 total 30.00 last 500.78) - open/close 19489 (time 1740485821.02 total 40.00 last 455.44) - open/close 24252 (time 1740485831.02 total 50.00 last 476.29) - open/close 29047 (time 1740485841.02 total 60.00 last 479.47) - open/close 30000 (time 1740485843.29 total 62.27 last 421.06) - open/close 34502 (time 1740485853.29 total 72.27 last 450.16) - open/close 39085 (time 1740485863.29 total 82.27 last 458.25) - open/close 43691 (time 1740485873.29 total 92.27 last 460.60) - open/close 48364 (time 1740485883.29 total 102.27 last 467.29) - open/close 50000 (time 1740485887.01 total 105.99 last 439.74) - open/close 54556 (time 1740485897.01 total 115.99 last 455.55) - open/close 59208 (time 1740485907.01 total 125.99 last 465.15) - open/close 64118 (time 1740485917.01 total 135.99 last 490.96) total: 64768 open/close in 137.39 seconds: 471.41 ops/second - /unlink 8569 (time 1740485929.29 total 10.00 last 856.84) - /unlink 17719 (time 1740485939.29 total 20.00 last 914.92) - /unlink 20000 (time 1740485941.87 total 22.58 last 882.99) - /unlink 28810 (time 1740485951.87 total 32.59 last 880.95) - /unlink 37582 (time 1740485961.87 total 42.59 last 877.17) - /unlink 40000 (time 1740485964.79 total 45.51 last 828.05) - /unlink 48656 (time 1740485974.79 total 55.51 last 865.59) - /unlink 57510 (time 1740485984.80 total 65.51 last 885.33) - /unlink 60000 (time 1740485987.46 total 68.17 last 936.20) total: 64768 /unlink in 73.59 seconds: 880.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5557 (time 1740486030.88 total 10.00 last 555.70) - open/close 10000 (time 1740486039.00 total 18.13 last 546.83) - open/close 15406 (time 1740486049.00 total 28.13 last 540.60) - open/close 20000 (time 1740486057.61 total 36.73 last 533.79) - open/close 25449 (time 1740486067.61 total 46.73 last 544.85) - open/close 30000 (time 1740486076.27 total 55.40 last 525.28) - open/close 35406 (time 1740486086.27 total 65.40 last 540.54) - open/close 40000 (time 1740486094.98 total 74.10 last 527.70) - open/close 45197 (time 1740486104.98 total 84.10 last 519.61) - open/close 50000 (time 1740486114.43 total 93.55 last 508.48) - open/close 55192 (time 1740486124.43 total 103.55 last 519.15) - open/close 60000 (time 1740486133.76 total 112.88 last 515.49) - open/close 64427 (time 1740486143.76 total 122.88 last 442.70) total: 64768 open/close in 123.59 seconds: 524.05 ops/second - /unlink 9233 (time 1740486155.29 total 10.00 last 923.26) - /unlink 18087 (time 1740486165.29 total 20.00 last 885.38) - /unlink 20000 (time 1740486167.41 total 22.12 last 900.91) - /unlink 29262 (time 1740486177.41 total 32.12 last 926.15) - /unlink 38010 (time 1740486187.41 total 42.13 last 874.77) - /unlink 40000 (time 1740486189.57 total 44.28 last 921.37) - /unlink 49027 (time 1740486199.57 total 54.29 last 902.68) - /unlink 57743 (time 1740486209.57 total 64.29 last 871.49) - /unlink 60000 (time 1740486212.02 total 66.74 last 920.99) total: 64768 /unlink in 71.89 seconds: 900.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2