== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:06:17 (1768140377) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) oleg332-server: oleg332-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg332-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3765 (time 1768140428.41 total 10.00 last 376.44) - open/close 7308 (time 1768140438.41 total 20.00 last 354.28) - open/close 10000 (time 1768140446.28 total 27.87 last 342.19) - open/close 13742 (time 1768140456.29 total 37.87 last 374.04) - open/close 17193 (time 1768140466.29 total 47.88 last 345.00) - open/close 20000 (time 1768140474.87 total 56.45 last 327.22) - open/close 23562 (time 1768140484.87 total 66.46 last 356.11) - open/close 25896 (time 1768140494.87 total 76.46 last 233.32) - open/close 28685 (time 1768140504.88 total 86.46 last 278.82) - open/close 30000 (time 1768140510.28 total 91.87 last 243.44) - open/close 33022 (time 1768140520.28 total 101.87 last 302.14) - open/close 35689 (time 1768140530.28 total 111.87 last 266.65) - open/close 38091 (time 1768140540.28 total 121.87 last 240.11) - open/close 40000 (time 1768140546.21 total 127.80 last 322.09) - open/close 43089 (time 1768140556.21 total 137.80 last 308.82) - open/close 45635 (time 1768140566.22 total 147.80 last 254.54) - open/close 48137 (time 1768140576.22 total 157.81 last 250.12) - open/close 50000 (time 1768140585.73 total 167.31 last 195.98) - open/close 52586 (time 1768140595.73 total 177.31 last 258.58) - open/close 55485 (time 1768140605.73 total 187.32 last 289.87) - open/close 58214 (time 1768140615.73 total 197.32 last 272.77) - open/close 60000 (time 1768140622.06 total 203.65 last 282.05) - open/close 63099 (time 1768140632.07 total 213.65 last 309.84) total: 64768 open/close in 219.03 seconds: 295.70 ops/second - /unlink 5538 (time 1768140649.00 total 10.00 last 553.57) - /unlink 10000 (time 1768140658.57 total 19.57 last 466.28) - /unlink 15321 (time 1768140668.57 total 29.57 last 532.05) - /unlink 20000 (time 1768140676.40 total 37.40 last 597.55) - /unlink 25915 (time 1768140686.40 total 47.41 last 591.46) - /unlink 30000 (time 1768140692.88 total 53.89 last 630.23) - /unlink 36017 (time 1768140702.89 total 63.89 last 601.62) - /unlink 40000 (time 1768140709.28 total 70.28 last 623.25) - /unlink 46351 (time 1768140719.28 total 80.28 last 635.04) - /unlink 50000 (time 1768140724.05 total 85.05 last 764.39) - /unlink 57308 (time 1768140734.05 total 95.06 last 730.71) - /unlink 60000 (time 1768140738.09 total 99.09 last 667.43) total: 64768 /unlink in 106.17 seconds: 610.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5530 (time 1768140779.81 total 10.00 last 553.00) - open/close 10000 (time 1768140788.54 total 18.73 last 512.20) - open/close 15354 (time 1768140798.54 total 28.73 last 535.33) - open/close 20000 (time 1768140807.07 total 37.26 last 544.71) - open/close 25124 (time 1768140817.07 total 47.26 last 512.33) - open/close 30000 (time 1768140826.15 total 56.34 last 537.03) - open/close 35158 (time 1768140836.15 total 66.34 last 515.78) - open/close 39986 (time 1768140846.15 total 76.34 last 482.72) - open/close 45063 (time 1768140856.15 total 86.34 last 507.65) - open/close 50000 (time 1768140865.25 total 95.44 last 542.84) - open/close 54727 (time 1768140875.25 total 105.44 last 472.63) - open/close 59252 (time 1768140885.25 total 115.44 last 452.41) - open/close 63959 (time 1768140895.25 total 125.44 last 470.62) total: 64768 open/close in 126.93 seconds: 510.25 ops/second - /unlink 9118 (time 1768140908.07 total 10.00 last 911.77) - /unlink 17808 (time 1768140918.07 total 20.00 last 868.94) - /unlink 20000 (time 1768140921.42 total 23.35 last 654.54) - /unlink 27832 (time 1768140931.42 total 33.35 last 783.18) - /unlink 30000 (time 1768140933.69 total 35.62 last 954.74) - /unlink 38137 (time 1768140943.69 total 45.62 last 813.69) - /unlink 40000 (time 1768140946.29 total 48.22 last 717.55) - /unlink 48743 (time 1768140956.29 total 58.22 last 874.29) - /unlink 57234 (time 1768140966.29 total 68.22 last 849.10) - /unlink 60000 (time 1768140969.15 total 71.08 last 967.64) total: 64768 /unlink in 76.25 seconds: 849.47 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4883 (time 1768141003.98 total 10.00 last 488.27) - open/close 9681 (time 1768141013.98 total 20.00 last 479.78) - open/close 14136 (time 1768141023.98 total 30.00 last 445.40) - open/close 18863 (time 1768141033.98 total 40.00 last 472.69) - open/close 20000 (time 1768141036.47 total 42.49 last 456.51) - open/close 24822 (time 1768141046.47 total 52.50 last 482.12) - open/close 29731 (time 1768141056.47 total 62.50 last 490.87) - open/close 34444 (time 1768141066.48 total 72.50 last 471.24) - open/close 39297 (time 1768141076.48 total 82.50 last 485.28) - open/close 43937 (time 1768141086.48 total 92.50 last 463.98) - open/close 48482 (time 1768141096.48 total 102.50 last 454.44) - open/close 50000 (time 1768141099.79 total 105.81 last 458.21) - open/close 54863 (time 1768141109.79 total 115.81 last 486.28) - open/close 60000 (time 1768141119.14 total 125.16 last 549.56) - open/close 64710 (time 1768141129.14 total 135.16 last 470.92) total: 64768 open/close in 135.30 seconds: 478.69 ops/second - /unlink 8759 (time 1768141140.62 total 10.00 last 875.86) - /unlink 17413 (time 1768141150.62 total 20.00 last 865.40) - /unlink 20000 (time 1768141153.45 total 22.82 last 916.60) - /unlink 28654 (time 1768141163.45 total 32.82 last 865.35) - /unlink 37496 (time 1768141173.45 total 42.82 last 884.13) - /unlink 40000 (time 1768141176.98 total 46.36 last 708.61) - /unlink 47582 (time 1768141186.98 total 56.36 last 758.17) - /unlink 50000 (time 1768141190.33 total 59.71 last 722.07) - /unlink 57679 (time 1768141200.33 total 69.71 last 767.81) - /unlink 60000 (time 1768141203.06 total 72.44 last 851.02) total: 64768 /unlink in 78.56 seconds: 824.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4925 (time 1768141241.71 total 10.00 last 492.45) - open/close 9682 (time 1768141251.71 total 20.00 last 475.67) - open/close 14464 (time 1768141261.71 total 30.00 last 478.09) - open/close 19434 (time 1768141271.71 total 40.01 last 496.92) - open/close 24401 (time 1768141281.71 total 50.01 last 496.68) - open/close 29214 (time 1768141291.71 total 60.01 last 481.23) - open/close 33758 (time 1768141301.71 total 70.01 last 454.36) - open/close 38236 (time 1768141311.71 total 80.01 last 447.75) - open/close 40000 (time 1768141315.35 total 83.65 last 485.11) - open/close 44870 (time 1768141325.35 total 93.65 last 486.95) - open/close 49487 (time 1768141335.35 total 103.65 last 461.61) - open/close 54198 (time 1768141345.35 total 113.65 last 471.06) - open/close 58804 (time 1768141355.36 total 123.65 last 460.47) - open/close 60000 (time 1768141357.73 total 126.03 last 502.98) - open/close 64670 (time 1768141367.74 total 136.03 last 466.98) total: 64768 open/close in 136.35 seconds: 475.00 ops/second - /unlink 8181 (time 1768141379.60 total 10.00 last 818.03) - /unlink 10000 (time 1768141381.88 total 12.28 last 798.93) - /unlink 18136 (time 1768141391.88 total 22.28 last 813.55) - /unlink 20000 (time 1768141395.10 total 25.50 last 577.73) - /unlink 27955 (time 1768141405.11 total 35.51 last 795.37) - /unlink 30000 (time 1768141407.91 total 38.31 last 729.97) - /unlink 37915 (time 1768141417.91 total 48.31 last 791.49) - /unlink 40000 (time 1768141420.46 total 50.86 last 815.73) - /unlink 47732 (time 1768141430.46 total 60.86 last 773.14) - /unlink 50000 (time 1768141433.53 total 63.93 last 740.90) - /unlink 57822 (time 1768141443.53 total 73.93 last 782.05) - /unlink 60000 (time 1768141446.12 total 76.52 last 840.35) total: 64768 /unlink in 82.29 seconds: 787.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4926 (time 1768141485.13 total 10.00 last 492.59) - open/close 10000 (time 1768141494.76 total 19.63 last 527.05) - open/close 15101 (time 1768141504.76 total 29.63 last 510.08) - open/close 20000 (time 1768141514.57 total 39.44 last 499.37) - open/close 24845 (time 1768141524.57 total 49.44 last 484.44) - open/close 29812 (time 1768141534.57 total 59.44 last 496.66) - open/close 34817 (time 1768141544.57 total 69.44 last 500.46) - open/close 39131 (time 1768141554.57 total 79.44 last 431.30) - open/close 43767 (time 1768141564.57 total 89.44 last 463.54) - open/close 46836 (time 1768141574.57 total 99.44 last 306.90) - open/close 50000 (time 1768141583.06 total 107.93 last 372.90) - open/close 54437 (time 1768141593.06 total 117.93 last 443.65) - open/close 58826 (time 1768141603.06 total 127.93 last 438.81) - open/close 60000 (time 1768141605.97 total 130.84 last 404.19) - open/close 64320 (time 1768141615.97 total 140.84 last 431.94) total: 64768 open/close in 141.83 seconds: 456.65 ops/second - /unlink 8123 (time 1768141628.43 total 10.00 last 812.25) - /unlink 10000 (time 1768141630.52 total 12.08 last 901.12) - /unlink 17925 (time 1768141640.52 total 22.08 last 792.49) - /unlink 20000 (time 1768141643.36 total 24.92 last 730.39) - /unlink 28116 (time 1768141653.36 total 34.93 last 811.32) - /unlink 30000 (time 1768141655.69 total 37.26 last 808.70) - /unlink 37883 (time 1768141665.69 total 47.26 last 788.23) - /unlink 40000 (time 1768141668.07 total 49.64 last 889.17) - /unlink 47698 (time 1768141678.07 total 59.64 last 769.75) - /unlink 50000 (time 1768141681.28 total 62.84 last 718.84) - /unlink 57953 (time 1768141691.28 total 72.84 last 795.26) - /unlink 60000 (time 1768141693.52 total 75.09 last 911.49) total: 64768 /unlink in 81.36 seconds: 796.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2