== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:13:14 (1722481994) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 6297 (time 1722482028.30 total 10.00 last 629.64) - open/close 10000 (time 1722482034.75 total 16.45 last 574.12) - open/close 15314 (time 1722482044.75 total 26.45 last 531.37) - open/close 20000 (time 1722482052.79 total 34.49 last 582.61) - open/close 25550 (time 1722482062.80 total 44.50 last 554.95) - open/close 30000 (time 1722482071.22 total 52.92 last 528.17) - open/close 35664 (time 1722482081.22 total 62.92 last 566.33) - open/close 40000 (time 1722482088.55 total 70.25 last 591.44) - open/close 45229 (time 1722482098.62 total 80.32 last 519.21) - open/close 50000 (time 1722482106.91 total 88.61 last 576.08) - open/close 55559 (time 1722482116.91 total 98.61 last 555.88) - open/close 60000 (time 1722482126.03 total 107.73 last 486.75) total: 64768 open/close in 117.26 seconds: 552.37 ops/second - /unlink 8353 (time 1722482146.40 total 10.00 last 835.24) - /unlink 16945 (time 1722482156.40 total 20.00 last 859.17) - /unlink 20000 (time 1722482160.10 total 23.70 last 826.46) - /unlink 28251 (time 1722482170.10 total 33.70 last 825.09) - /unlink 30000 (time 1722482172.39 total 35.99 last 763.85) - /unlink 38426 (time 1722482182.39 total 45.99 last 842.55) - /unlink 47036 (time 1722482192.39 total 55.99 last 860.94) - /unlink 50000 (time 1722482195.89 total 59.49 last 847.49) - /unlink 58665 (time 1722482205.89 total 69.49 last 866.45) total: 64768 /unlink in 76.12 seconds: 850.81 ops/second - open/close 5237 (time 1722482223.37 total 10.00 last 523.63) - open/close 10000 (time 1722482232.47 total 19.10 last 523.48) - open/close 15176 (time 1722482242.47 total 29.10 last 517.59) - open/close 20000 (time 1722482252.26 total 38.89 last 492.81) - open/close 25099 (time 1722482262.26 total 48.89 last 509.82) - open/close 29966 (time 1722482272.26 total 58.89 last 486.68) - open/close 35046 (time 1722482282.26 total 68.89 last 507.94) - open/close 40000 (time 1722482292.10 total 78.73 last 503.75) - open/close 45211 (time 1722482302.10 total 88.73 last 521.02) - open/close 50000 (time 1722482311.67 total 98.30 last 500.08) - open/close 54951 (time 1722482321.68 total 108.31 last 495.06) - open/close 59156 (time 1722482331.68 total 118.31 last 420.46) - open/close 64196 (time 1722482341.68 total 128.31 last 503.97) total: 64768 open/close in 129.55 seconds: 499.94 ops/second - /unlink 8909 (time 1722482354.04 total 10.00 last 890.85) - /unlink 18076 (time 1722482364.04 total 20.00 last 916.64) - /unlink 20000 (time 1722482366.18 total 22.15 last 896.06) - /unlink 29085 (time 1722482376.18 total 32.15 last 908.47) - /unlink 37819 (time 1722482386.18 total 42.15 last 873.37) - /unlink 40000 (time 1722482388.43 total 44.40 last 969.41) - /unlink 49249 (time 1722482398.43 total 54.40 last 924.80) - /unlink 58547 (time 1722482408.44 total 64.40 last 929.72) total: 64768 /unlink in 71.09 seconds: 911.07 ops/second - open/close 5457 (time 1722482426.21 total 10.00 last 545.67) - open/close 10000 (time 1722482435.13 total 18.92 last 509.55) - open/close 15371 (time 1722482445.13 total 28.92 last 537.02) - open/close 20000 (time 1722482453.78 total 37.57 last 534.98) - open/close 25329 (time 1722482463.78 total 47.57 last 532.87) - open/close 30000 (time 1722482472.82 total 56.61 last 516.77) - open/close 34485 (time 1722482482.82 total 66.61 last 448.44) - open/close 39752 (time 1722482492.82 total 76.61 last 526.60) - open/close 45013 (time 1722482502.88 total 86.67 last 523.27) - open/close 49710 (time 1722482512.88 total 96.67 last 469.62) - open/close 54425 (time 1722482522.88 total 106.67 last 471.43) - open/close 58438 (time 1722482532.88 total 116.67 last 401.29) - open/close 60000 (time 1722482536.55 total 120.34 last 426.24) - open/close 63536 (time 1722482546.55 total 130.34 last 353.57) total: 64768 open/close in 133.11 seconds: 486.57 ops/second - /unlink 8506 (time 1722482560.14 total 10.00 last 850.48) - /unlink 17704 (time 1722482570.14 total 20.00 last 919.74) - /unlink 20000 (time 1722482572.62 total 22.48 last 925.09) - /unlink 29057 (time 1722482582.62 total 32.48 last 905.68) - /unlink 37381 (time 1722482592.63 total 42.49 last 832.26) - /unlink 40000 (time 1722482595.95 total 45.81 last 788.49) - /unlink 48824 (time 1722482605.95 total 55.81 last 882.30) - /unlink 57672 (time 1722482615.95 total 65.81 last 884.76) - /unlink 60000 (time 1722482618.52 total 68.38 last 904.36) total: 64768 /unlink in 73.51 seconds: 881.13 ops/second - open/close 4477 (time 1722482634.65 total 10.00 last 447.67) - open/close 8761 (time 1722482644.65 total 20.00 last 428.33) - open/close 10000 (time 1722482647.20 total 22.55 last 485.93) - open/close 14227 (time 1722482657.20 total 32.55 last 422.65) - open/close 19374 (time 1722482667.20 total 42.55 last 514.70) - open/close 24367 (time 1722482677.20 total 52.55 last 499.26) - open/close 29260 (time 1722482687.20 total 62.55 last 489.27) - open/close 33433 (time 1722482697.20 total 72.56 last 417.28) - open/close 38505 (time 1722482707.20 total 82.56 last 507.13) - open/close 40000 (time 1722482710.71 total 86.07 last 426.11) - open/close 44774 (time 1722482720.71 total 96.07 last 477.31) - open/close 48308 (time 1722482730.72 total 106.07 last 353.26) - open/close 50000 (time 1722482736.89 total 112.24 last 274.06) - open/close 53958 (time 1722482746.89 total 122.25 last 395.79) - open/close 57961 (time 1722482756.89 total 132.25 last 400.23) - open/close 60000 (time 1722482761.23 total 136.58 last 470.45) - open/close 63420 (time 1722482771.27 total 146.63 last 340.33) total: 64768 open/close in 149.55 seconds: 433.08 ops/second - /unlink 6902 (time 1722482785.11 total 10.00 last 690.14) - /unlink 10000 (time 1722482789.27 total 14.16 last 745.68) - /unlink 17222 (time 1722482799.27 total 24.16 last 722.12) - /unlink 20000 (time 1722482803.79 total 28.67 last 614.83) - /unlink 25244 (time 1722482813.79 total 38.68 last 524.36) - /unlink 30000 (time 1722482821.45 total 46.34 last 620.77) - /unlink 35291 (time 1722482831.45 total 56.34 last 529.03) - /unlink 40000 (time 1722482840.03 total 64.92 last 548.79) - /unlink 46169 (time 1722482850.03 total 74.92 last 616.82) - /unlink 50000 (time 1722482856.40 total 81.29 last 601.22) - /unlink 55677 (time 1722482866.40 total 91.29 last 567.69) - /unlink 60000 (time 1722482873.12 total 98.01 last 643.45) total: 64768 /unlink in 107.50 seconds: 602.49 ops/second - open/close 2640 (time 1722482893.85 total 10.00 last 263.97) - open/close 5297 (time 1722482903.85 total 20.00 last 265.66) - open/close 8242 (time 1722482913.85 total 30.00 last 294.46) - open/close 10000 (time 1722482919.19 total 35.35 last 328.95) - open/close 13865 (time 1722482929.19 total 45.35 last 386.49) - open/close 17942 (time 1722482939.20 total 55.35 last 407.65) - open/close 20000 (time 1722482945.56 total 61.72 last 323.32) - open/close 22522 (time 1722482955.56 total 71.72 last 252.19) - open/close 25189 (time 1722482965.56 total 81.72 last 266.69) - open/close 27871 (time 1722482975.56 total 91.72 last 268.19) - open/close 30000 (time 1722482984.08 total 100.23 last 249.92) - open/close 32393 (time 1722482994.08 total 110.23 last 239.30) - open/close 34974 (time 1722483004.08 total 120.24 last 258.01) - open/close 37544 (time 1722483014.09 total 130.24 last 256.95) - open/close 40000 (time 1722483023.52 total 139.68 last 260.28) - open/close 42566 (time 1722483033.53 total 149.68 last 256.51) - open/close 44997 (time 1722483043.53 total 159.68 last 243.03) - open/close 47370 (time 1722483053.53 total 169.69 last 237.23) - open/close 49940 (time 1722483063.53 total 179.69 last 256.99) - open/close 52332 (time 1722483073.53 total 189.69 last 239.18) - open/close 54739 (time 1722483083.54 total 199.69 last 240.63) - open/close 57318 (time 1722483093.54 total 209.69 last 257.88) - open/close 59634 (time 1722483103.60 total 219.76 last 230.05) - open/close 62072 (time 1722483113.61 total 229.76 last 243.74) - open/close 64692 (time 1722483123.61 total 239.76 last 261.92) total: 64768 open/close in 240.06 seconds: 269.80 ops/second - /unlink 4550 (time 1722483135.16 total 10.00 last 454.91) - /unlink 9276 (time 1722483145.16 total 20.00 last 472.52) - /unlink 14735 (time 1722483155.16 total 30.00 last 545.89) - /unlink 19271 (time 1722483165.16 total 40.00 last 453.56) - /unlink 23791 (time 1722483175.16 total 50.01 last 451.94) - /unlink 28743 (time 1722483185.16 total 60.01 last 495.13) - /unlink 30000 (time 1722483187.66 total 62.51 last 503.26) - /unlink 34751 (time 1722483197.66 total 72.51 last 475.01) - /unlink 39884 (time 1722483207.66 total 82.51 last 513.18) - /unlink 45458 (time 1722483217.66 total 92.51 last 557.40) - /unlink 50000 (time 1722483226.74 total 101.58 last 500.58) - /unlink 54782 (time 1722483236.74 total 111.58 last 478.16) - /unlink 59297 (time 1722483246.74 total 121.58 last 451.46) - /unlink 64032 (time 1722483256.74 total 131.58 last 473.48) total: 64768 /unlink in 132.89 seconds: 487.38 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server LNET ready to unload unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: LNET ready to unload modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2