== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:46:30 (1746625590) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg151-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5481 (time 1746625624.24 total 10.00 last 548.04) - open/close 10000 (time 1746625632.79 total 18.55 last 528.31) - open/close 15208 (time 1746625642.79 total 28.56 last 520.72) - open/close 20000 (time 1746625651.53 total 37.30 last 548.28) - open/close 25158 (time 1746625661.53 total 47.30 last 515.79) - open/close 28355 (time 1746625671.54 total 57.30 last 319.65) - open/close 29681 (time 1746625681.55 total 67.31 last 132.44) - open/close 30000 (time 1746625684.57 total 70.33 last 105.57) - open/close 31382 (time 1746625694.57 total 80.34 last 138.14) - open/close 33613 (time 1746625704.58 total 90.34 last 222.98) - open/close 36393 (time 1746625714.58 total 100.34 last 277.92) - open/close 38514 (time 1746625724.59 total 110.35 last 211.98) - open/close 40000 (time 1746625733.43 total 119.19 last 168.03) - open/close 42699 (time 1746625743.43 total 129.19 last 269.89) - open/close 46673 (time 1746625753.43 total 139.20 last 397.31) - open/close 50000 (time 1746625760.88 total 146.64 last 446.83) - open/close 52907 (time 1746625770.88 total 156.64 last 290.69) - open/close 55607 (time 1746625780.88 total 166.64 last 269.97) - open/close 58549 (time 1746625790.88 total 176.65 last 294.15) - open/close 60000 (time 1746625795.22 total 180.98 last 334.86) - open/close 64089 (time 1746625805.22 total 190.98 last 408.83) total: 64768 open/close in 193.39 seconds: 334.90 ops/second - /unlink 4950 (time 1746625818.58 total 10.00 last 494.98) - /unlink 10000 (time 1746625825.17 total 16.59 last 766.22) - /unlink 17066 (time 1746625835.17 total 26.59 last 706.57) - /unlink 20000 (time 1746625839.40 total 30.82 last 693.57) - /unlink 26983 (time 1746625849.40 total 40.82 last 698.26) - /unlink 30000 (time 1746625853.23 total 44.65 last 788.79) - /unlink 39418 (time 1746625863.23 total 54.65 last 941.74) - /unlink 48834 (time 1746625873.23 total 64.65 last 941.58) - /unlink 58227 (time 1746625883.23 total 74.65 last 939.23) - /unlink 60000 (time 1746625885.26 total 76.68 last 874.44) total: 64768 /unlink in 82.38 seconds: 786.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5137 (time 1746625921.89 total 10.00 last 513.61) - open/close 9860 (time 1746625931.90 total 20.00 last 472.18) - open/close 13392 (time 1746625941.90 total 30.01 last 353.14) - open/close 18437 (time 1746625951.90 total 40.01 last 504.50) - open/close 20000 (time 1746625955.03 total 43.13 last 499.57) - open/close 24781 (time 1746625965.03 total 53.14 last 478.01) - open/close 29561 (time 1746625975.03 total 63.14 last 477.98) - open/close 34107 (time 1746625985.03 total 73.14 last 454.53) - open/close 38905 (time 1746625995.03 total 83.14 last 479.79) - open/close 44488 (time 1746626005.03 total 93.14 last 558.18) - open/close 49574 (time 1746626015.04 total 103.14 last 508.56) - open/close 54180 (time 1746626025.04 total 113.14 last 460.59) - open/close 59023 (time 1746626035.04 total 123.14 last 484.16) - open/close 60000 (time 1746626037.93 total 126.04 last 337.75) total: 64768 open/close in 135.47 seconds: 478.11 ops/second - /unlink 8758 (time 1746626058.16 total 10.00 last 875.71) - /unlink 16291 (time 1746626068.16 total 20.00 last 753.22) - /unlink 20000 (time 1746626073.49 total 25.33 last 695.97) - /unlink 26001 (time 1746626083.49 total 35.33 last 600.05) - /unlink 30000 (time 1746626090.32 total 42.17 last 585.06) - /unlink 36399 (time 1746626100.32 total 52.17 last 639.82) - /unlink 40000 (time 1746626106.67 total 58.51 last 567.70) - /unlink 46043 (time 1746626116.67 total 68.51 last 604.24) - /unlink 50000 (time 1746626123.43 total 75.28 last 584.99) - /unlink 58170 (time 1746626133.43 total 85.28 last 816.98) - /unlink 60000 (time 1746626135.51 total 87.35 last 881.27) total: 64768 /unlink in 94.50 seconds: 685.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3711 (time 1746626173.26 total 10.00 last 371.03) - open/close 7185 (time 1746626183.26 total 20.00 last 347.37) - open/close 10000 (time 1746626190.70 total 27.44 last 378.68) - open/close 13607 (time 1746626200.70 total 37.44 last 360.59) - open/close 17744 (time 1746626210.70 total 47.44 last 413.70) - open/close 20000 (time 1746626214.41 total 51.15 last 607.77) - open/close 24066 (time 1746626224.42 total 61.16 last 406.41) - open/close 28039 (time 1746626234.42 total 71.16 last 397.24) - open/close 30000 (time 1746626238.03 total 74.77 last 542.60) - open/close 34777 (time 1746626248.03 total 84.77 last 477.69) - open/close 39576 (time 1746626258.03 total 94.77 last 479.84) - open/close 43431 (time 1746626268.04 total 104.78 last 385.42) - open/close 47069 (time 1746626278.04 total 114.78 last 363.75) - open/close 50000 (time 1746626283.91 total 120.65 last 499.21) - open/close 55137 (time 1746626293.91 total 130.65 last 513.64) - open/close 59538 (time 1746626303.91 total 140.65 last 440.04) - open/close 63154 (time 1746626313.92 total 150.65 last 361.48) total: 64768 open/close in 155.76 seconds: 415.83 ops/second - /unlink 6373 (time 1746626330.02 total 10.00 last 637.29) - /unlink 10000 (time 1746626334.77 total 14.75 last 764.25) - /unlink 17630 (time 1746626344.77 total 24.75 last 762.94) - /unlink 20000 (time 1746626348.58 total 28.56 last 620.99) - /unlink 26441 (time 1746626358.59 total 38.56 last 644.07) - /unlink 30000 (time 1746626363.55 total 43.53 last 716.21) - /unlink 37530 (time 1746626373.56 total 53.53 last 752.97) - /unlink 40000 (time 1746626376.71 total 56.69 last 782.78) - /unlink 48089 (time 1746626386.71 total 66.69 last 808.84) - /unlink 50000 (time 1746626389.11 total 69.09 last 797.62) - /unlink 58128 (time 1746626399.11 total 79.09 last 812.74) - /unlink 60000 (time 1746626401.27 total 81.25 last 865.27) total: 64768 /unlink in 86.48 seconds: 748.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5232 (time 1746626438.19 total 10.00 last 523.08) - open/close 10000 (time 1746626446.48 total 18.29 last 575.22) - open/close 15754 (time 1746626456.48 total 28.29 last 575.38) - open/close 20000 (time 1746626463.81 total 35.63 last 578.87) - open/close 25922 (time 1746626473.81 total 45.63 last 592.14) - open/close 30000 (time 1746626480.51 total 52.32 last 609.01) - open/close 35507 (time 1746626490.51 total 62.32 last 550.63) - open/close 40000 (time 1746626499.53 total 71.35 last 497.97) - open/close 45462 (time 1746626509.53 total 81.35 last 546.18) - open/close 50000 (time 1746626517.22 total 89.04 last 590.27) - open/close 55964 (time 1746626527.22 total 99.04 last 596.40) - open/close 60000 (time 1746626534.15 total 105.96 last 582.71) total: 64768 open/close in 114.17 seconds: 567.32 ops/second - /unlink 8413 (time 1746626553.13 total 10.00 last 841.26) - /unlink 17223 (time 1746626563.13 total 20.00 last 880.96) - /unlink 20000 (time 1746626566.22 total 23.08 last 900.58) - /unlink 28902 (time 1746626576.22 total 33.08 last 890.16) - /unlink 37456 (time 1746626586.22 total 43.09 last 855.31) - /unlink 40000 (time 1746626589.21 total 46.08 last 848.68) - /unlink 48574 (time 1746626599.22 total 56.08 last 857.33) - /unlink 56898 (time 1746626609.22 total 66.08 last 832.40) - /unlink 60000 (time 1746626612.87 total 69.74 last 848.82) total: 64768 /unlink in 75.61 seconds: 856.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5013 (time 1746626653.75 total 10.00 last 501.21) - open/close 10000 (time 1746626663.68 total 19.93 last 502.22) - open/close 15201 (time 1746626673.68 total 29.94 last 519.87) - open/close 20000 (time 1746626683.59 total 39.84 last 484.53) - open/close 22388 (time 1746626693.59 total 49.84 last 238.76) - open/close 24651 (time 1746626703.59 total 59.84 last 226.27) - open/close 28538 (time 1746626713.59 total 69.84 last 388.69) - open/close 30000 (time 1746626716.56 total 72.81 last 492.98) - open/close 34288 (time 1746626726.56 total 82.81 last 428.74) - open/close 37183 (time 1746626736.56 total 92.81 last 289.45) - open/close 40000 (time 1746626743.97 total 100.22 last 380.43) - open/close 44564 (time 1746626753.97 total 110.22 last 456.38) - open/close 48992 (time 1746626763.97 total 120.22 last 442.73) - open/close 50000 (time 1746626766.17 total 122.42 last 457.81) - open/close 54933 (time 1746626776.17 total 132.42 last 493.23) - open/close 57832 (time 1746626786.17 total 142.42 last 289.83) - open/close 60000 (time 1746626793.81 total 150.06 last 283.95) - open/close 63879 (time 1746626803.81 total 160.06 last 387.87) total: 64768 open/close in 162.48 seconds: 398.62 ops/second - /unlink 10000 (time 1746626816.98 total 9.88 last 1011.91) - /unlink 19187 (time 1746626826.98 total 19.88 last 918.68) - /unlink 28277 (time 1746626836.98 total 29.88 last 908.93) - /unlink 36566 (time 1746626846.98 total 39.88 last 828.85) - /unlink 40000 (time 1746626851.21 total 44.11 last 811.77) - /unlink 48669 (time 1746626861.21 total 54.12 last 866.83) - /unlink 56759 (time 1746626871.21 total 64.12 last 808.96) - /unlink 60000 (time 1746626875.04 total 67.94 last 847.55) total: 64768 /unlink in 73.31 seconds: 883.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2