== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:17:36 (1737130656) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg227-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2152 (time 1737130695.72 total 10.00 last 215.18) - open/close 4133 (time 1737130705.72 total 20.00 last 198.08) - open/close 6675 (time 1737130715.72 total 30.00 last 254.14) - open/close 9448 (time 1737130725.72 total 40.01 last 277.26) - open/close 10000 (time 1737130727.85 total 42.13 last 259.23) - open/close 12923 (time 1737130737.86 total 52.14 last 292.23) - open/close 15302 (time 1737130747.86 total 62.14 last 237.81) - open/close 17707 (time 1737130757.86 total 72.14 last 240.49) - open/close 20000 (time 1737130765.32 total 79.60 last 307.51) - open/close 23223 (time 1737130775.32 total 89.60 last 322.25) - open/close 26571 (time 1737130785.32 total 99.60 last 334.63) - open/close 29099 (time 1737130795.33 total 109.61 last 252.64) - open/close 30000 (time 1737130799.57 total 113.85 last 212.65) - open/close 32814 (time 1737130809.57 total 123.85 last 281.34) - open/close 35908 (time 1737130819.57 total 133.85 last 309.27) - open/close 39152 (time 1737130829.58 total 143.86 last 324.32) - open/close 40000 (time 1737130832.50 total 146.78 last 289.91) - open/close 42504 (time 1737130842.50 total 156.79 last 250.32) - open/close 44968 (time 1737130852.50 total 166.79 last 246.39) - open/close 47345 (time 1737130862.51 total 176.79 last 237.63) - open/close 50000 (time 1737130870.87 total 185.15 last 317.42) - open/close 52416 (time 1737130880.88 total 195.16 last 241.49) - open/close 54828 (time 1737130890.88 total 205.16 last 241.17) - open/close 57496 (time 1737130900.88 total 215.16 last 266.76) - open/close 60000 (time 1737130910.73 total 225.01 last 254.27) - open/close 62370 (time 1737130920.73 total 235.01 last 236.95) total: 64768 open/close in 243.32 seconds: 266.19 ops/second - /unlink 4600 (time 1737130940.11 total 10.00 last 459.87) - /unlink 8521 (time 1737130950.11 total 20.00 last 392.03) - /unlink 10000 (time 1737130953.17 total 23.06 last 484.00) - /unlink 14424 (time 1737130963.17 total 33.06 last 442.34) - /unlink 19495 (time 1737130973.17 total 43.06 last 507.08) - /unlink 24237 (time 1737130983.17 total 53.06 last 474.10) - /unlink 28054 (time 1737130993.17 total 63.07 last 381.67) - /unlink 30000 (time 1737130997.17 total 67.06 last 486.51) - /unlink 34708 (time 1737131007.17 total 77.07 last 470.77) - /unlink 39512 (time 1737131017.17 total 87.07 last 480.25) - /unlink 43041 (time 1737131027.18 total 97.07 last 352.81) - /unlink 47406 (time 1737131037.18 total 107.07 last 436.45) - /unlink 50000 (time 1737131043.32 total 113.22 last 422.10) - /unlink 53731 (time 1737131053.32 total 123.22 last 373.03) - /unlink 57600 (time 1737131063.33 total 133.22 last 386.78) - /unlink 60000 (time 1737131069.51 total 139.40 last 388.43) - /unlink 64665 (time 1737131079.51 total 149.40 last 466.42) total: 64768 /unlink in 149.64 seconds: 432.83 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2984 (time 1737131108.99 total 10.00 last 298.38) - open/close 5317 (time 1737131119.00 total 20.01 last 233.18) - open/close 7854 (time 1737131129.00 total 30.01 last 253.66) - open/close 10000 (time 1737131137.37 total 38.38 last 256.30) - open/close 12571 (time 1737131147.37 total 48.38 last 257.06) - open/close 15125 (time 1737131157.37 total 58.38 last 255.39) - open/close 18294 (time 1737131167.38 total 68.39 last 316.79) - open/close 20000 (time 1737131173.85 total 74.86 last 263.49) - open/close 22769 (time 1737131183.85 total 84.86 last 276.85) - open/close 25497 (time 1737131193.86 total 94.86 last 272.74) - open/close 28264 (time 1737131203.86 total 104.87 last 276.67) - open/close 30000 (time 1737131209.12 total 110.13 last 329.81) - open/close 32785 (time 1737131219.12 total 120.13 last 278.42) - open/close 35119 (time 1737131229.12 total 130.13 last 233.40) - open/close 37677 (time 1737131239.13 total 140.14 last 255.58) - open/close 40000 (time 1737131248.55 total 149.56 last 246.59) - open/close 42870 (time 1737131258.56 total 159.56 last 286.92) - open/close 45610 (time 1737131268.56 total 169.56 last 273.99) - open/close 48798 (time 1737131278.56 total 179.57 last 318.75) - open/close 50000 (time 1737131283.12 total 184.13 last 263.61) - open/close 52852 (time 1737131293.12 total 194.13 last 285.16) - open/close 55913 (time 1737131303.12 total 204.13 last 306.06) - open/close 59083 (time 1737131313.12 total 214.13 last 317.00) - open/close 63561 (time 1737131323.12 total 224.13 last 447.74) total: 64768 open/close in 226.84 seconds: 285.53 ops/second - /unlink 8997 (time 1737131336.69 total 10.00 last 899.60) - /unlink 18574 (time 1737131346.69 total 20.00 last 957.64) - /unlink 26971 (time 1737131356.69 total 30.00 last 839.66) - /unlink 30000 (time 1737131360.43 total 33.74 last 809.66) - /unlink 37698 (time 1737131370.44 total 43.74 last 769.73) - /unlink 40000 (time 1737131373.02 total 46.33 last 891.25) - /unlink 48882 (time 1737131383.02 total 56.33 last 888.16) - /unlink 58974 (time 1737131393.02 total 66.33 last 1009.19) total: 64768 /unlink in 71.91 seconds: 900.72 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4583 (time 1737131440.00 total 10.00 last 458.22) - open/close 9056 (time 1737131450.00 total 20.00 last 447.24) - open/close 13189 (time 1737131460.00 total 30.00 last 413.26) - open/close 17836 (time 1737131470.00 total 40.00 last 464.67) - open/close 20000 (time 1737131474.28 total 44.29 last 505.20) - open/close 25006 (time 1737131484.29 total 54.29 last 500.54) - open/close 30000 (time 1737131494.28 total 64.29 last 499.52) - open/close 34718 (time 1737131504.28 total 74.29 last 471.77) - open/close 40000 (time 1737131514.24 total 84.24 last 530.53) - open/close 45386 (time 1737131524.24 total 94.24 last 538.59) - open/close 49623 (time 1737131534.24 total 104.25 last 423.56) - open/close 53994 (time 1737131544.24 total 114.25 last 437.09) - open/close 58379 (time 1737131554.24 total 124.25 last 438.42) - open/close 60000 (time 1737131557.59 total 127.60 last 484.23) total: 64768 open/close in 136.51 seconds: 474.44 ops/second - /unlink 8102 (time 1737131577.32 total 10.00 last 810.09) - /unlink 10000 (time 1737131579.48 total 12.16 last 879.36) - /unlink 18456 (time 1737131589.48 total 22.16 last 845.57) - /unlink 27327 (time 1737131599.48 total 32.16 last 887.07) - /unlink 30000 (time 1737131602.91 total 35.59 last 780.33) - /unlink 36477 (time 1737131612.91 total 45.59 last 647.66) - /unlink 40000 (time 1737131618.15 total 50.83 last 671.87) - /unlink 48160 (time 1737131628.15 total 60.83 last 815.99) - /unlink 50000 (time 1737131630.18 total 62.86 last 908.12) - /unlink 58693 (time 1737131640.18 total 72.86 last 869.13) total: 64768 /unlink in 79.56 seconds: 814.13 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4291 (time 1737131680.78 total 10.00 last 429.02) - open/close 8109 (time 1737131690.78 total 20.00 last 381.79) - open/close 10000 (time 1737131695.73 total 24.96 last 381.79) - open/close 14459 (time 1737131705.73 total 34.96 last 445.84) - open/close 18617 (time 1737131715.73 total 44.96 last 415.78) - open/close 20000 (time 1737131719.68 total 48.90 last 350.30) - open/close 24144 (time 1737131729.68 total 58.91 last 414.35) - open/close 28590 (time 1737131739.68 total 68.91 last 444.53) - open/close 30000 (time 1737131742.62 total 71.85 last 480.00) - open/close 34979 (time 1737131752.62 total 81.85 last 497.89) - open/close 39042 (time 1737131762.62 total 91.85 last 406.25) - open/close 43908 (time 1737131772.62 total 101.85 last 486.56) - open/close 48479 (time 1737131782.62 total 111.85 last 457.02) - open/close 50000 (time 1737131785.83 total 115.05 last 475.15) - open/close 55414 (time 1737131795.83 total 125.05 last 541.35) - open/close 60000 (time 1737131804.43 total 133.65 last 533.32) total: 64768 open/close in 142.65 seconds: 454.02 ops/second - /unlink 9793 (time 1737131824.21 total 10.00 last 979.23) - /unlink 19494 (time 1737131834.21 total 20.00 last 970.01) - /unlink 27181 (time 1737131844.21 total 30.00 last 768.62) - /unlink 30000 (time 1737131848.22 total 34.02 last 702.56) - /unlink 39239 (time 1737131858.22 total 44.02 last 923.83) - /unlink 49070 (time 1737131868.23 total 54.02 last 982.98) - /unlink 57769 (time 1737131878.23 total 64.02 last 869.82) - /unlink 60000 (time 1737131880.65 total 66.44 last 920.16) total: 64768 /unlink in 71.89 seconds: 900.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4969 (time 1737131921.40 total 10.00 last 496.89) - open/close 9926 (time 1737131931.40 total 20.00 last 495.69) - open/close 15021 (time 1737131941.40 total 30.00 last 509.43) - open/close 20000 (time 1737131950.80 total 39.40 last 529.88) - open/close 25669 (time 1737131960.80 total 49.40 last 566.83) - open/close 30000 (time 1737131969.44 total 58.04 last 501.21) - open/close 35038 (time 1737131979.44 total 68.04 last 503.79) - open/close 39718 (time 1737131989.44 total 78.04 last 467.94) - open/close 44476 (time 1737131999.44 total 88.04 last 475.73) - open/close 49102 (time 1737132009.44 total 98.04 last 462.57) - open/close 53961 (time 1737132019.44 total 108.04 last 485.88) - open/close 58850 (time 1737132029.44 total 118.05 last 488.89) - open/close 60000 (time 1737132031.46 total 120.06 last 571.21) total: 64768 open/close in 128.21 seconds: 505.18 ops/second - /unlink 10000 (time 1737132050.35 total 9.95 last 1004.63) - /unlink 19917 (time 1737132060.35 total 19.95 last 991.61) - /unlink 29386 (time 1737132070.35 total 29.95 last 946.90) - /unlink 38072 (time 1737132080.36 total 39.96 last 868.36) - /unlink 40000 (time 1737132082.50 total 42.10 last 901.18) - /unlink 49026 (time 1737132092.50 total 52.10 last 902.60) - /unlink 58266 (time 1737132102.50 total 62.10 last 923.91) total: 64768 /unlink in 69.08 seconds: 937.55 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server LNET ready to unload unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: LNET ready to unload modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2