== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:13:28 (1759842808) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg216-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock oleg216-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4797 (time 1759842855.07 total 10.00 last 479.59) - open/close 8812 (time 1759842865.08 total 20.00 last 401.43) - open/close 10000 (time 1759842867.66 total 22.59 last 459.26) - open/close 14231 (time 1759842877.66 total 32.59 last 423.08) - open/close 18886 (time 1759842887.66 total 42.59 last 465.42) - open/close 20000 (time 1759842889.90 total 44.83 last 498.77) - open/close 24770 (time 1759842899.90 total 54.83 last 476.93) - open/close 28771 (time 1759842909.90 total 64.83 last 400.04) - open/close 30000 (time 1759842913.34 total 68.27 last 357.02) - open/close 34389 (time 1759842923.34 total 78.27 last 438.84) - open/close 39009 (time 1759842933.35 total 88.27 last 461.95) - open/close 43920 (time 1759842943.35 total 98.28 last 491.02) - open/close 48359 (time 1759842953.35 total 108.28 last 443.88) - open/close 50000 (time 1759842957.25 total 112.18 last 420.08) - open/close 54602 (time 1759842967.26 total 122.18 last 460.16) - open/close 59108 (time 1759842977.26 total 132.18 last 450.56) - open/close 63549 (time 1759842987.26 total 142.19 last 444.04) total: 64768 open/close in 144.89 seconds: 447.01 ops/second - /unlink 6596 (time 1759843001.25 total 10.00 last 659.56) - /unlink 10000 (time 1759843006.94 total 15.68 last 599.07) - /unlink 16826 (time 1759843016.94 total 25.68 last 682.55) - /unlink 20000 (time 1759843021.52 total 30.26 last 692.88) - /unlink 27401 (time 1759843031.52 total 40.26 last 740.08) - /unlink 30000 (time 1759843035.19 total 43.94 last 707.24) - /unlink 36759 (time 1759843045.19 total 53.94 last 675.88) - /unlink 40000 (time 1759843049.65 total 58.40 last 727.30) - /unlink 47015 (time 1759843059.65 total 68.40 last 701.48) - /unlink 50000 (time 1759843063.55 total 72.30 last 764.46) - /unlink 58156 (time 1759843073.56 total 82.30 last 815.52) - /unlink 60000 (time 1759843075.91 total 84.65 last 784.26) total: 64768 /unlink in 90.65 seconds: 714.47 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4623 (time 1759843114.41 total 10.00 last 462.20) - open/close 8930 (time 1759843124.42 total 20.00 last 430.58) - open/close 10000 (time 1759843126.71 total 22.30 last 466.25) - open/close 14661 (time 1759843136.71 total 32.30 last 466.02) - open/close 18966 (time 1759843146.71 total 42.30 last 430.43) - open/close 20000 (time 1759843149.03 total 44.61 last 447.36) - open/close 24239 (time 1759843159.03 total 54.62 last 423.86) - open/close 28302 (time 1759843169.03 total 64.62 last 406.23) - open/close 30000 (time 1759843173.02 total 68.61 last 424.86) - open/close 34360 (time 1759843183.02 total 78.61 last 435.99) - open/close 38741 (time 1759843193.03 total 88.62 last 438.01) - open/close 40000 (time 1759843196.13 total 91.72 last 405.24) - open/close 44174 (time 1759843206.13 total 101.72 last 417.37) - open/close 48368 (time 1759843216.13 total 111.72 last 419.39) - open/close 50000 (time 1759843219.91 total 115.50 last 431.96) - open/close 54289 (time 1759843229.91 total 125.50 last 428.85) - open/close 58565 (time 1759843239.92 total 135.50 last 427.52) - open/close 60000 (time 1759843243.33 total 138.92 last 420.74) - open/close 64179 (time 1759843253.33 total 148.92 last 417.81) total: 64768 open/close in 150.43 seconds: 430.56 ops/second - /unlink 6653 (time 1759843266.28 total 10.00 last 665.29) - /unlink 10000 (time 1759843271.24 total 14.96 last 674.98) - /unlink 17008 (time 1759843281.24 total 24.96 last 700.77) - /unlink 20000 (time 1759843285.33 total 29.05 last 732.10) - /unlink 26924 (time 1759843295.36 total 39.08 last 689.87) - /unlink 30000 (time 1759843299.68 total 43.40 last 712.85) - /unlink 37191 (time 1759843309.68 total 53.40 last 719.02) - /unlink 40000 (time 1759843313.78 total 57.50 last 684.79) - /unlink 46978 (time 1759843323.78 total 67.50 last 697.79) - /unlink 50000 (time 1759843328.08 total 71.79 last 703.88) - /unlink 57030 (time 1759843338.08 total 81.80 last 702.95) - /unlink 60000 (time 1759843342.32 total 86.03 last 700.60) total: 64768 /unlink in 92.42 seconds: 700.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4666 (time 1759843380.95 total 10.00 last 466.56) - open/close 9290 (time 1759843390.95 total 20.00 last 462.37) - open/close 13826 (time 1759843400.95 total 30.00 last 453.59) - open/close 18523 (time 1759843410.96 total 40.00 last 469.62) - open/close 20000 (time 1759843414.09 total 43.13 last 472.05) - open/close 24507 (time 1759843424.09 total 53.13 last 450.61) - open/close 28962 (time 1759843434.09 total 63.13 last 445.48) - open/close 30000 (time 1759843436.57 total 65.61 last 418.99) - open/close 34477 (time 1759843446.57 total 75.61 last 447.63) - open/close 39033 (time 1759843456.57 total 85.62 last 455.49) - open/close 40000 (time 1759843458.71 total 87.75 last 452.54) - open/close 44535 (time 1759843468.71 total 97.75 last 453.42) - open/close 48756 (time 1759843478.71 total 107.76 last 422.04) - open/close 50000 (time 1759843481.56 total 110.61 last 435.85) - open/close 54251 (time 1759843491.56 total 120.61 last 425.06) - open/close 58662 (time 1759843501.57 total 130.61 last 441.03) - open/close 60000 (time 1759843504.60 total 133.64 last 441.31) - open/close 64408 (time 1759843514.60 total 143.65 last 440.73) total: 64768 open/close in 144.56 seconds: 448.03 ops/second - /unlink 7299 (time 1759843526.90 total 10.00 last 729.85) - /unlink 10000 (time 1759843530.23 total 13.33 last 812.08) - /unlink 17266 (time 1759843540.23 total 23.33 last 726.58) - /unlink 20000 (time 1759843544.19 total 27.28 last 690.85) - /unlink 26012 (time 1759843554.19 total 37.28 last 601.19) - /unlink 30000 (time 1759843560.23 total 43.33 last 659.66) - /unlink 36892 (time 1759843570.23 total 53.33 last 689.16) - /unlink 40000 (time 1759843574.75 total 57.84 last 688.50) - /unlink 47072 (time 1759843584.75 total 67.85 last 707.13) - /unlink 50000 (time 1759843588.87 total 71.97 last 710.25) - /unlink 56694 (time 1759843598.87 total 81.97 last 669.33) - /unlink 60000 (time 1759843603.48 total 86.58 last 716.81) total: 64768 /unlink in 93.14 seconds: 695.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2998 (time 1759843641.77 total 10.01 last 299.65) - open/close 4217 (time 1759843651.77 total 20.01 last 121.87) - open/close 5307 (time 1759843661.80 total 30.04 last 108.70) - open/close 8790 (time 1759843671.80 total 40.04 last 348.22) - open/close 10000 (time 1759843674.51 total 42.74 last 447.66) - open/close 14792 (time 1759843684.51 total 52.74 last 479.19) - open/close 19953 (time 1759843694.51 total 62.74 last 516.00) - open/close 24607 (time 1759843704.51 total 72.74 last 465.38) - open/close 29488 (time 1759843714.51 total 82.74 last 488.08) - open/close 34481 (time 1759843724.51 total 92.75 last 499.19) - open/close 39070 (time 1759843734.55 total 102.78 last 457.13) - open/close 40000 (time 1759843736.65 total 104.88 last 443.73) - open/close 44274 (time 1759843746.65 total 114.88 last 427.32) - open/close 48280 (time 1759843756.65 total 124.88 last 400.54) - open/close 50000 (time 1759843760.80 total 129.03 last 414.54) - open/close 54327 (time 1759843770.80 total 139.03 last 432.67) - open/close 58768 (time 1759843780.80 total 149.03 last 444.08) - open/close 60000 (time 1759843783.35 total 151.59 last 482.78) total: 64768 open/close in 161.46 seconds: 401.14 ops/second - /unlink 6850 (time 1759843804.64 total 10.00 last 684.93) - /unlink 10000 (time 1759843808.68 total 14.04 last 780.54) - /unlink 16822 (time 1759843818.68 total 24.04 last 682.11) - /unlink 20000 (time 1759843823.34 total 28.70 last 681.53) - /unlink 26905 (time 1759843833.35 total 38.70 last 690.36) - /unlink 30000 (time 1759843838.12 total 43.48 last 647.73) - /unlink 37130 (time 1759843848.12 total 53.48 last 712.97) - /unlink 40000 (time 1759843853.78 total 59.14 last 507.18) - /unlink 45337 (time 1759843863.78 total 69.14 last 533.63) - /unlink 50000 (time 1759843872.20 total 77.55 last 554.34) - /unlink 56701 (time 1759843882.20 total 87.55 last 670.09) - /unlink 60000 (time 1759843887.31 total 92.67 last 645.04) total: 64768 /unlink in 99.37 seconds: 651.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4018 (time 1759843922.68 total 10.00 last 401.76) - open/close 7470 (time 1759843932.68 total 20.00 last 345.12) - open/close 10000 (time 1759843940.13 total 27.45 last 339.54) - open/close 13457 (time 1759843950.13 total 37.45 last 345.70) - open/close 17328 (time 1759843960.13 total 47.45 last 387.09) - open/close 20000 (time 1759843967.39 total 54.72 last 367.89) - open/close 23641 (time 1759843977.40 total 64.72 last 364.02) - open/close 27209 (time 1759843987.40 total 74.72 last 356.74) - open/close 30000 (time 1759843993.95 total 81.27 last 426.02) - open/close 34419 (time 1759844003.95 total 91.27 last 441.88) - open/close 39076 (time 1759844013.95 total 101.27 last 465.63) - open/close 40000 (time 1759844015.97 total 103.29 last 458.14) - open/close 44633 (time 1759844025.97 total 113.29 last 463.25) - open/close 48763 (time 1759844035.97 total 123.29 last 412.93) - open/close 50000 (time 1759844039.31 total 126.63 last 370.77) - open/close 53477 (time 1759844049.31 total 136.63 last 347.67) - open/close 57887 (time 1759844059.31 total 146.63 last 440.95) - open/close 60000 (time 1759844064.09 total 151.42 last 441.78) - open/close 64324 (time 1759844074.09 total 161.42 last 432.39) total: 64768 open/close in 162.48 seconds: 398.62 ops/second - /unlink 7118 (time 1759844086.44 total 10.00 last 711.80) - /unlink 10000 (time 1759844090.36 total 13.92 last 735.27) - /unlink 17416 (time 1759844100.36 total 23.92 last 741.52) - /unlink 20000 (time 1759844103.63 total 27.20 last 788.42) - /unlink 27489 (time 1759844113.64 total 37.20 last 748.83) - /unlink 30000 (time 1759844117.18 total 40.74 last 708.42) - /unlink 37057 (time 1759844127.18 total 50.74 last 705.68) - /unlink 40000 (time 1759844131.92 total 55.49 last 620.58) - /unlink 46748 (time 1759844141.92 total 65.49 last 674.79) - /unlink 50000 (time 1759844146.30 total 69.86 last 743.10) - /unlink 56456 (time 1759844156.30 total 79.86 last 645.56) - /unlink 60000 (time 1759844161.38 total 84.94 last 697.72) total: 64768 /unlink in 91.79 seconds: 705.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg216-server: LNET ready to unload oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2