== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:14:43 (1715202883) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock oleg339-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4049 (time 1715202919.76 total 10.00 last 404.81) - open/close 7425 (time 1715202929.76 total 20.00 last 337.58) - open/close 10000 (time 1715202936.02 total 26.26 last 411.48) - open/close 14187 (time 1715202946.02 total 36.26 last 418.62) - open/close 18401 (time 1715202956.02 total 46.26 last 421.37) - open/close 20000 (time 1715202959.69 total 49.94 last 435.31) - open/close 24018 (time 1715202969.69 total 59.94 last 401.76) - open/close 28621 (time 1715202979.69 total 69.94 last 460.30) - open/close 30000 (time 1715202981.98 total 72.23 last 602.75) - open/close 35726 (time 1715202991.98 total 82.23 last 572.51) - open/close 40000 (time 1715203000.13 total 90.37 last 524.68) - open/close 44995 (time 1715203010.13 total 100.37 last 499.41) - open/close 49865 (time 1715203020.13 total 110.38 last 486.98) - open/close 54322 (time 1715203030.13 total 120.38 last 445.64) - open/close 58512 (time 1715203040.13 total 130.38 last 418.94) - open/close 60000 (time 1715203043.61 total 133.86 last 427.84) - open/close 61811 (time 1715203053.61 total 143.86 last 181.07) - open/close 62971 (time 1715203063.62 total 153.87 last 115.90) total: 64768 open/close in 162.28 seconds: 399.11 ops/second - /unlink 4138 (time 1715203083.07 total 10.00 last 413.78) - /unlink 8225 (time 1715203093.07 total 20.00 last 408.58) - /unlink 10000 (time 1715203097.74 total 24.67 last 380.00) - /unlink 14003 (time 1715203107.74 total 34.68 last 400.25) - /unlink 20000 (time 1715203116.54 total 43.47 last 682.12) - /unlink 28308 (time 1715203126.54 total 53.47 last 830.75) - /unlink 37080 (time 1715203136.54 total 63.47 last 877.07) - /unlink 40000 (time 1715203140.70 total 67.63 last 702.27) - /unlink 48538 (time 1715203150.70 total 77.63 last 853.75) - /unlink 50000 (time 1715203153.38 total 80.32 last 543.92) - /unlink 55235 (time 1715203163.39 total 90.32 last 523.44) - /unlink 60000 (time 1715203172.11 total 99.04 last 546.00) - /unlink 64493 (time 1715203182.11 total 109.05 last 449.26) total: 64768 /unlink in 109.88 seconds: 589.44 ops/second - open/close 2274 (time 1715203194.13 total 10.00 last 227.38) - open/close 4489 (time 1715203204.13 total 20.00 last 221.46) - open/close 7347 (time 1715203214.13 total 30.00 last 285.77) - open/close 10000 (time 1715203222.84 total 38.71 last 304.67) - open/close 12840 (time 1715203232.85 total 48.71 last 283.92) - open/close 15952 (time 1715203242.85 total 58.72 last 311.14) - open/close 19571 (time 1715203252.85 total 68.72 last 361.84) - open/close 23090 (time 1715203262.85 total 78.72 last 351.87) - open/close 26744 (time 1715203272.85 total 88.72 last 365.37) - open/close 29956 (time 1715203282.85 total 98.72 last 321.10) - open/close 33153 (time 1715203292.85 total 108.72 last 319.67) - open/close 36482 (time 1715203302.86 total 118.73 last 332.81) - open/close 39472 (time 1715203312.86 total 128.73 last 298.99) - open/close 43383 (time 1715203322.86 total 138.73 last 391.09) - open/close 47197 (time 1715203332.86 total 148.73 last 381.39) - open/close 50000 (time 1715203340.59 total 156.46 last 362.53) - open/close 53448 (time 1715203350.59 total 166.46 last 344.75) - open/close 56598 (time 1715203360.59 total 176.46 last 314.93) - open/close 60000 (time 1715203370.24 total 186.11 last 352.70) - open/close 63188 (time 1715203380.24 total 196.11 last 318.74) total: 64768 open/close in 201.30 seconds: 321.74 ops/second - /unlink 5583 (time 1715203396.41 total 10.00 last 558.26) - /unlink 10000 (time 1715203404.19 total 17.78 last 567.46) - /unlink 17037 (time 1715203414.19 total 27.78 last 703.68) - /unlink 20000 (time 1715203418.36 total 31.95 last 711.02) - /unlink 27475 (time 1715203428.36 total 41.95 last 747.46) - /unlink 30000 (time 1715203432.08 total 45.67 last 678.88) - /unlink 36761 (time 1715203442.08 total 55.67 last 676.01) - /unlink 40000 (time 1715203446.33 total 59.92 last 763.28) - /unlink 46785 (time 1715203456.33 total 69.92 last 678.42) - /unlink 50000 (time 1715203460.24 total 73.84 last 820.57) - /unlink 57259 (time 1715203470.25 total 83.84 last 725.84) - /unlink 60000 (time 1715203473.89 total 87.48 last 752.14) total: 64768 /unlink in 94.14 seconds: 688.02 ops/second - open/close 4231 (time 1715203491.45 total 10.00 last 423.06) - open/close 8349 (time 1715203501.45 total 20.00 last 411.75) - open/close 10000 (time 1715203505.68 total 24.24 last 389.87) - open/close 14068 (time 1715203515.69 total 34.24 last 406.74) - open/close 19049 (time 1715203525.69 total 44.24 last 498.10) - open/close 20000 (time 1715203527.75 total 46.30 last 461.41) - open/close 25175 (time 1715203537.75 total 56.30 last 517.43) - open/close 29833 (time 1715203547.75 total 66.30 last 465.72) - open/close 34022 (time 1715203557.75 total 76.30 last 418.89) - open/close 38153 (time 1715203567.75 total 86.30 last 413.08) - open/close 40000 (time 1715203572.22 total 90.77 last 413.37) - open/close 44683 (time 1715203582.22 total 100.77 last 468.15) - open/close 49391 (time 1715203592.22 total 110.78 last 470.73) - open/close 53195 (time 1715203602.23 total 120.78 last 380.33) - open/close 56747 (time 1715203612.23 total 130.78 last 355.15) - open/close 60000 (time 1715203620.29 total 138.85 last 403.22) - open/close 64443 (time 1715203630.30 total 148.85 last 444.22) total: 64768 open/close in 149.72 seconds: 432.59 ops/second - /unlink 8667 (time 1715203642.07 total 10.00 last 866.67) - /unlink 17420 (time 1715203652.07 total 20.00 last 875.25) - /unlink 20000 (time 1715203655.12 total 23.06 last 844.25) - /unlink 28132 (time 1715203665.12 total 33.06 last 813.20) - /unlink 30000 (time 1715203667.55 total 35.49 last 768.61) - /unlink 38581 (time 1715203677.55 total 45.49 last 858.07) - /unlink 47014 (time 1715203687.55 total 55.49 last 843.29) - /unlink 50000 (time 1715203691.39 total 59.32 last 778.40) - /unlink 58472 (time 1715203701.39 total 69.32 last 847.14) total: 64768 /unlink in 76.23 seconds: 849.67 ops/second - open/close 5495 (time 1715203719.17 total 10.00 last 549.47) - open/close 10000 (time 1715203727.80 total 18.64 last 521.56) - open/close 15509 (time 1715203737.80 total 28.64 last 550.89) - open/close 20000 (time 1715203747.54 total 38.37 last 461.39) - open/close 25320 (time 1715203757.54 total 48.37 last 531.94) - open/close 30000 (time 1715203766.72 total 57.55 last 509.82) - open/close 35294 (time 1715203776.72 total 67.55 last 529.31) - open/close 40000 (time 1715203786.36 total 77.19 last 488.24) - open/close 44828 (time 1715203796.36 total 87.19 last 482.74) - open/close 49827 (time 1715203806.36 total 97.19 last 499.89) - open/close 55282 (time 1715203816.36 total 107.20 last 545.47) - open/close 60000 (time 1715203824.58 total 115.41 last 574.07) total: 64768 open/close in 123.58 seconds: 524.10 ops/second - /unlink 9198 (time 1715203843.55 total 10.00 last 919.74) - /unlink 19041 (time 1715203853.55 total 20.00 last 984.27) - /unlink 28113 (time 1715203863.55 total 30.00 last 907.15) - /unlink 30000 (time 1715203865.59 total 32.05 last 922.05) - /unlink 38984 (time 1715203875.59 total 42.05 last 898.33) - /unlink 48544 (time 1715203885.60 total 52.05 last 955.95) - /unlink 58105 (time 1715203895.60 total 62.05 last 956.01) total: 64768 /unlink in 68.48 seconds: 945.84 ops/second - open/close 5887 (time 1715203912.93 total 10.00 last 588.67) - open/close 10000 (time 1715203920.91 total 17.98 last 515.59) - open/close 14714 (time 1715203930.91 total 27.98 last 471.33) - open/close 19966 (time 1715203940.91 total 37.98 last 525.16) - open/close 25675 (time 1715203950.91 total 47.98 last 570.81) - open/close 30000 (time 1715203958.86 total 55.94 last 543.69) - open/close 35197 (time 1715203968.87 total 65.94 last 519.59) - open/close 40000 (time 1715203977.53 total 74.60 last 554.61) - open/close 45489 (time 1715203987.53 total 84.60 last 548.84) - open/close 50000 (time 1715203995.07 total 92.14 last 598.29) - open/close 55548 (time 1715204005.07 total 102.14 last 554.78) - open/close 60000 (time 1715204014.17 total 111.24 last 489.12) total: 64768 open/close in 119.30 seconds: 542.91 ops/second - /unlink 10000 (time 1715204032.45 total 9.46 last 1057.23) - /unlink 20000 (time 1715204042.09 total 19.10 last 1037.46) - /unlink 28781 (time 1715204052.09 total 29.10 last 878.06) - /unlink 37307 (time 1715204062.09 total 39.10 last 852.53) - /unlink 40000 (time 1715204065.25 total 42.25 last 853.64) - /unlink 48569 (time 1715204075.25 total 52.25 last 856.85) - /unlink 57622 (time 1715204085.25 total 62.25 last 905.25) - /unlink 60000 (time 1715204087.75 total 64.76 last 948.53) total: 64768 /unlink in 69.69 seconds: 929.38 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server LNET ready to unload unloading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: LNET ready to unload modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2