== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:54:37 (1772304877) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) oleg347-server: oleg347-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg347-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4362 (time 1772304921.63 total 10.00 last 436.10) - open/close 8931 (time 1772304931.63 total 20.00 last 456.88) - open/close 10000 (time 1772304934.12 total 22.49 last 429.68) - open/close 13885 (time 1772304944.12 total 32.49 last 388.50) - open/close 17664 (time 1772304954.12 total 42.49 last 377.83) - open/close 20000 (time 1772304959.50 total 47.88 last 433.72) - open/close 24717 (time 1772304969.51 total 57.88 last 471.64) - open/close 29477 (time 1772304979.51 total 67.88 last 475.92) - open/close 34482 (time 1772304989.51 total 77.88 last 500.40) - open/close 38360 (time 1772304999.51 total 87.89 last 387.73) - open/close 40000 (time 1772305003.10 total 91.47 last 457.38) - open/close 44232 (time 1772305013.10 total 101.47 last 423.15) - open/close 48874 (time 1772305023.10 total 111.47 last 464.18) - open/close 50000 (time 1772305025.38 total 113.76 last 492.81) - open/close 54461 (time 1772305035.38 total 123.76 last 446.09) - open/close 58773 (time 1772305045.38 total 133.76 last 431.17) - open/close 60000 (time 1772305048.15 total 136.53 last 443.38) - open/close 64316 (time 1772305058.15 total 146.53 last 431.55) total: 64768 open/close in 147.62 seconds: 438.75 ops/second - /unlink 6992 (time 1772305070.64 total 10.00 last 699.14) - /unlink 10000 (time 1772305074.79 total 14.16 last 723.84) - /unlink 16636 (time 1772305084.79 total 24.16 last 663.59) - /unlink 20000 (time 1772305089.66 total 29.02 last 691.67) - /unlink 26932 (time 1772305099.66 total 39.02 last 693.12) - /unlink 30000 (time 1772305104.52 total 43.88 last 631.45) - /unlink 35869 (time 1772305114.52 total 53.88 last 586.90) - /unlink 40000 (time 1772305121.51 total 60.88 last 590.40) - /unlink 45563 (time 1772305131.52 total 70.88 last 556.25) - /unlink 50000 (time 1772305138.88 total 78.24 last 602.62) - /unlink 55526 (time 1772305148.88 total 88.24 last 552.53) - /unlink 60000 (time 1772305156.24 total 95.61 last 607.59) total: 64768 /unlink in 103.68 seconds: 624.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3894 (time 1772305196.78 total 10.00 last 389.37) - open/close 7543 (time 1772305206.78 total 20.00 last 364.85) - open/close 10000 (time 1772305213.24 total 26.46 last 380.63) - open/close 13488 (time 1772305223.24 total 36.46 last 348.73) - open/close 17554 (time 1772305233.24 total 46.46 last 406.55) - open/close 20000 (time 1772305241.72 total 54.94 last 288.46) - open/close 23077 (time 1772305251.72 total 64.94 last 307.65) - open/close 26708 (time 1772305261.72 total 74.94 last 363.05) - open/close 30000 (time 1772305271.06 total 84.28 last 352.42) - open/close 33265 (time 1772305281.06 total 94.28 last 326.49) - open/close 36394 (time 1772305291.07 total 104.29 last 312.87) - open/close 39363 (time 1772305301.07 total 114.29 last 296.77) - open/close 42824 (time 1772305311.07 total 124.29 last 346.03) - open/close 45460 (time 1772305321.07 total 134.29 last 263.56) - open/close 48370 (time 1772305331.07 total 144.30 last 290.95) - open/close 50000 (time 1772305336.24 total 149.46 last 315.55) - open/close 53029 (time 1772305346.24 total 159.46 last 302.87) - open/close 56168 (time 1772305356.24 total 169.46 last 313.89) - open/close 58946 (time 1772305366.24 total 179.46 last 277.77) - open/close 60000 (time 1772305370.07 total 183.29 last 275.41) - open/close 63405 (time 1772305380.07 total 193.29 last 340.41) total: 64768 open/close in 197.00 seconds: 328.78 ops/second - /unlink 5988 (time 1772305395.11 total 10.00 last 598.69) - /unlink 10000 (time 1772305403.00 total 17.89 last 508.32) - /unlink 16075 (time 1772305413.01 total 27.90 last 607.41) - /unlink 20000 (time 1772305418.65 total 33.54 last 695.54) - /unlink 27211 (time 1772305428.65 total 43.54 last 720.90) - /unlink 30000 (time 1772305432.26 total 47.15 last 772.46) - /unlink 36947 (time 1772305442.26 total 57.15 last 694.63) - /unlink 40000 (time 1772305446.88 total 61.77 last 661.30) - /unlink 47152 (time 1772305456.88 total 71.77 last 715.13) - /unlink 50000 (time 1772305460.89 total 75.78 last 710.17) - /unlink 57312 (time 1772305470.89 total 85.78 last 731.16) - /unlink 60000 (time 1772305474.21 total 89.10 last 809.02) total: 64768 /unlink in 95.83 seconds: 675.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4892 (time 1772305512.99 total 10.00 last 489.18) - open/close 9839 (time 1772305522.99 total 20.00 last 494.69) - open/close 15173 (time 1772305532.99 total 30.00 last 533.37) - open/close 20000 (time 1772305542.23 total 39.25 last 522.16) - open/close 25154 (time 1772305552.23 total 49.25 last 515.37) - open/close 30000 (time 1772305561.35 total 58.36 last 531.48) - open/close 35271 (time 1772305571.35 total 68.36 last 527.04) - open/close 40000 (time 1772305580.78 total 77.79 last 501.48) - open/close 44465 (time 1772305590.79 total 87.80 last 446.40) - open/close 48892 (time 1772305600.79 total 97.80 last 442.60) - open/close 50000 (time 1772305603.30 total 100.31 last 441.73) - open/close 54417 (time 1772305613.30 total 110.31 last 441.67) - open/close 58689 (time 1772305623.30 total 120.31 last 427.18) - open/close 60000 (time 1772305626.10 total 123.11 last 468.47) - open/close 64511 (time 1772305636.10 total 133.11 last 451.10) total: 64768 open/close in 133.79 seconds: 484.09 ops/second - /unlink 7816 (time 1772305648.20 total 10.00 last 781.58) - /unlink 10000 (time 1772305651.04 total 12.85 last 767.25) - /unlink 18477 (time 1772305661.04 total 22.85 last 847.66) - /unlink 26627 (time 1772305671.05 total 32.85 last 814.91) - /unlink 30000 (time 1772305674.55 total 36.35 last 962.16) - /unlink 38396 (time 1772305684.55 total 46.35 last 839.52) - /unlink 40000 (time 1772305686.77 total 48.58 last 722.46) - /unlink 48540 (time 1772305696.77 total 58.58 last 853.98) - /unlink 57016 (time 1772305706.77 total 68.58 last 847.53) - /unlink 60000 (time 1772305709.93 total 71.73 last 945.18) total: 64768 /unlink in 77.23 seconds: 838.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4865 (time 1772305749.25 total 10.00 last 486.49) - open/close 9691 (time 1772305759.25 total 20.00 last 482.51) - open/close 14357 (time 1772305769.25 total 30.00 last 466.59) - open/close 19156 (time 1772305779.25 total 40.00 last 479.82) - open/close 23941 (time 1772305789.25 total 50.01 last 478.45) - open/close 28686 (time 1772305799.25 total 60.01 last 474.49) - open/close 30000 (time 1772305802.08 total 62.83 last 464.39) - open/close 34700 (time 1772305812.09 total 72.84 last 469.92) - open/close 39467 (time 1772305822.09 total 82.84 last 476.62) - open/close 44096 (time 1772305832.09 total 92.84 last 462.85) - open/close 48588 (time 1772305842.09 total 102.84 last 449.12) - open/close 50000 (time 1772305845.19 total 105.94 last 455.38) - open/close 54407 (time 1772305855.19 total 115.94 last 440.70) - open/close 58942 (time 1772305865.19 total 125.94 last 453.42) - open/close 60000 (time 1772305867.37 total 128.12 last 485.18) - open/close 64305 (time 1772305877.37 total 138.13 last 430.45) total: 64768 open/close in 139.28 seconds: 465.01 ops/second - /unlink 7119 (time 1772305889.85 total 10.00 last 711.86) - /unlink 10000 (time 1772305893.35 total 13.50 last 823.20) - /unlink 17215 (time 1772305903.35 total 23.50 last 721.47) - /unlink 20000 (time 1772305907.47 total 27.62 last 676.18) - /unlink 27160 (time 1772305917.47 total 37.62 last 715.92) - /unlink 30000 (time 1772305921.40 total 41.55 last 722.48) - /unlink 37418 (time 1772305931.40 total 51.55 last 741.74) - /unlink 40000 (time 1772305934.31 total 54.46 last 888.97) - /unlink 47219 (time 1772305944.31 total 64.46 last 721.86) - /unlink 50000 (time 1772305948.21 total 68.36 last 712.51) - /unlink 57826 (time 1772305958.21 total 78.36 last 782.55) - /unlink 60000 (time 1772305961.01 total 81.15 last 778.18) total: 64768 /unlink in 87.33 seconds: 741.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4139 (time 1772305999.45 total 10.00 last 413.79) - open/close 7376 (time 1772306009.45 total 20.01 last 323.62) - open/close 10000 (time 1772306017.65 total 28.20 last 320.17) - open/close 13434 (time 1772306027.65 total 38.20 last 343.37) - open/close 17382 (time 1772306037.65 total 48.20 last 394.73) - open/close 20000 (time 1772306044.85 total 55.40 last 363.80) - open/close 23258 (time 1772306054.85 total 65.40 last 325.77) - open/close 27205 (time 1772306064.85 total 75.40 last 394.60) - open/close 30000 (time 1772306072.65 total 83.20 last 358.41) - open/close 33821 (time 1772306082.65 total 93.20 last 381.99) - open/close 37169 (time 1772306092.65 total 103.20 last 334.78) - open/close 40000 (time 1772306101.50 total 112.05 last 320.05) - open/close 42765 (time 1772306111.50 total 122.05 last 276.46) - open/close 45338 (time 1772306121.50 total 132.05 last 257.25) - open/close 48199 (time 1772306131.50 total 142.05 last 286.08) - open/close 50000 (time 1772306137.04 total 147.59 last 325.33) - open/close 53300 (time 1772306147.04 total 157.59 last 329.89) - open/close 56146 (time 1772306157.04 total 167.60 last 284.50) - open/close 58735 (time 1772306167.05 total 177.60 last 258.84) - open/close 60000 (time 1772306170.83 total 181.38 last 334.38) - open/close 62872 (time 1772306180.83 total 191.39 last 287.09) total: 64768 open/close in 199.25 seconds: 325.06 ops/second - /unlink 6266 (time 1772306200.28 total 10.00 last 626.56) - /unlink 10000 (time 1772306205.05 total 14.77 last 783.63) - /unlink 15898 (time 1772306215.05 total 24.77 last 589.76) - /unlink 20000 (time 1772306220.72 total 30.44 last 722.85) - /unlink 26145 (time 1772306230.72 total 40.44 last 614.46) - /unlink 30000 (time 1772306236.31 total 46.03 last 689.34) - /unlink 37999 (time 1772306246.32 total 56.04 last 799.80) - /unlink 40000 (time 1772306249.23 total 58.95 last 685.54) - /unlink 47026 (time 1772306259.23 total 68.95 last 702.58) - /unlink 50000 (time 1772306262.66 total 72.38 last 868.33) - /unlink 57146 (time 1772306272.66 total 82.38 last 714.59) - /unlink 60000 (time 1772306276.20 total 85.92 last 805.69) total: 64768 /unlink in 93.01 seconds: 696.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg347-server: LNET ready to unload oleg347-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2