== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:34:45 (1767839685) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) oleg427-server: oleg427-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg427-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1905 (time 1767839741.76 total 10.00 last 190.42) - open/close 4139 (time 1767839751.76 total 20.01 last 223.23) - open/close 6098 (time 1767839761.77 total 30.01 last 195.87) - open/close 7952 (time 1767839771.77 total 40.02 last 185.33) - open/close 9714 (time 1767839781.77 total 50.02 last 176.16) - open/close 11767 (time 1767839791.77 total 60.02 last 205.25) - open/close 13885 (time 1767839801.78 total 70.03 last 211.72) - open/close 15981 (time 1767839811.78 total 80.03 last 209.60) - open/close 18211 (time 1767839821.78 total 90.03 last 222.93) - open/close 19868 (time 1767839831.79 total 100.04 last 165.60) - open/close 21456 (time 1767839841.79 total 110.04 last 158.74) - open/close 22973 (time 1767839851.80 total 120.04 last 151.65) - open/close 25022 (time 1767839861.80 total 130.04 last 204.87) - open/close 27011 (time 1767839871.80 total 140.05 last 198.84) - open/close 28846 (time 1767839881.80 total 150.05 last 183.43) - open/close 30000 (time 1767839890.24 total 158.49 last 136.73) - open/close 31626 (time 1767839900.24 total 168.49 last 162.57) - open/close 33425 (time 1767839910.24 total 178.49 last 179.90) - open/close 34807 (time 1767839920.25 total 188.49 last 138.19) - open/close 36413 (time 1767839930.25 total 198.50 last 160.48) - open/close 37964 (time 1767839940.25 total 208.50 last 155.10) - open/close 39374 (time 1767839950.26 total 218.51 last 140.87) - open/close 40000 (time 1767839954.11 total 222.36 last 162.51) - open/close 41620 (time 1767839964.12 total 232.37 last 161.95) - open/close 43166 (time 1767839974.12 total 242.37 last 154.58) - open/close 44911 (time 1767839984.13 total 252.37 last 174.39) - open/close 46516 (time 1767839994.13 total 262.37 last 160.46) - open/close 48382 (time 1767840004.13 total 272.38 last 186.58) - open/close 50000 (time 1767840012.00 total 280.25 last 205.61) - open/close 51781 (time 1767840022.00 total 290.25 last 178.06) - open/close 53839 (time 1767840032.00 total 300.25 last 205.76) - open/close 55798 (time 1767840042.00 total 310.25 last 195.89) - open/close 57566 (time 1767840052.01 total 320.25 last 176.75) - open/close 59162 (time 1767840062.01 total 330.26 last 159.48) - open/close 60000 (time 1767840067.24 total 335.48 last 160.47) - open/close 61634 (time 1767840077.25 total 345.50 last 163.12) - open/close 63062 (time 1767840087.26 total 355.50 last 142.77) total: 64768 open/close in 363.93 seconds: 177.97 ops/second - /unlink 2929 (time 1767840107.46 total 10.00 last 292.86) - /unlink 5584 (time 1767840117.46 total 20.00 last 265.50) - /unlink 8398 (time 1767840127.46 total 30.00 last 281.39) - /unlink 10000 (time 1767840133.36 total 35.90 last 271.84) - /unlink 12996 (time 1767840143.36 total 45.90 last 299.57) - /unlink 16125 (time 1767840153.36 total 55.90 last 312.87) - /unlink 19443 (time 1767840163.36 total 65.90 last 331.75) - /unlink 22043 (time 1767840173.36 total 75.90 last 259.98) - /unlink 24805 (time 1767840183.36 total 85.90 last 276.12) - /unlink 27900 (time 1767840193.36 total 95.90 last 309.48) - /unlink 30000 (time 1767840201.11 total 103.65 last 270.95) - /unlink 33037 (time 1767840211.12 total 113.66 last 303.60) - /unlink 36236 (time 1767840221.12 total 123.66 last 319.86) - /unlink 39057 (time 1767840231.12 total 133.66 last 281.97) - /unlink 40000 (time 1767840234.28 total 136.82 last 298.77) - /unlink 43275 (time 1767840244.28 total 146.82 last 327.45) - /unlink 46316 (time 1767840254.28 total 156.82 last 304.00) - /unlink 49183 (time 1767840264.28 total 166.82 last 286.69) - /unlink 50000 (time 1767840267.36 total 169.90 last 265.82) - /unlink 52720 (time 1767840277.37 total 179.91 last 271.74) - /unlink 54982 (time 1767840287.37 total 189.91 last 226.17) - /unlink 57520 (time 1767840297.37 total 199.91 last 253.75) - /unlink 60000 (time 1767840305.41 total 207.95 last 308.44) - /unlink 64010 (time 1767840315.41 total 217.95 last 400.95) total: 64768 /unlink in 219.61 seconds: 294.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1984 (time 1767840347.65 total 10.00 last 198.31) - open/close 4116 (time 1767840357.66 total 20.01 last 213.14) - open/close 5974 (time 1767840367.66 total 30.01 last 185.75) - open/close 7752 (time 1767840377.66 total 40.01 last 177.78) - open/close 10000 (time 1767840387.44 total 49.80 last 229.72) - open/close 14400 (time 1767840397.44 total 59.80 last 439.99) - open/close 18828 (time 1767840407.46 total 69.82 last 441.94) - open/close 20000 (time 1767840410.03 total 72.38 last 457.57) - open/close 24277 (time 1767840420.03 total 82.38 last 427.66) - open/close 28881 (time 1767840430.03 total 92.38 last 460.35) - open/close 30000 (time 1767840432.05 total 94.40 last 554.32) - open/close 35063 (time 1767840442.05 total 104.40 last 506.29) - open/close 39976 (time 1767840452.05 total 114.40 last 491.27) - open/close 44865 (time 1767840462.05 total 124.40 last 488.81) - open/close 49772 (time 1767840472.14 total 134.49 last 486.41) - open/close 55126 (time 1767840482.14 total 144.49 last 535.36) - open/close 60000 (time 1767840492.00 total 154.35 last 494.19) - open/close 64509 (time 1767840502.00 total 164.35 last 450.88) total: 64768 open/close in 164.98 seconds: 392.57 ops/second - /unlink 8258 (time 1767840513.91 total 10.00 last 825.75) - /unlink 16417 (time 1767840523.91 total 20.00 last 815.76) - /unlink 20000 (time 1767840528.65 total 24.73 last 757.39) - /unlink 27787 (time 1767840538.68 total 34.77 last 776.04) - /unlink 30000 (time 1767840541.33 total 37.42 last 835.36) - /unlink 38545 (time 1767840551.33 total 47.42 last 854.45) - /unlink 46453 (time 1767840561.33 total 57.42 last 790.79) - /unlink 50000 (time 1767840565.69 total 61.78 last 813.87) - /unlink 57894 (time 1767840575.69 total 71.78 last 789.35) - /unlink 60000 (time 1767840578.05 total 74.14 last 890.08) total: 64768 /unlink in 79.58 seconds: 813.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5087 (time 1767840618.17 total 10.00 last 508.66) - open/close 10000 (time 1767840627.01 total 18.84 last 556.03) - open/close 15347 (time 1767840637.01 total 28.84 last 534.66) - open/close 20000 (time 1767840645.68 total 37.51 last 536.42) - open/close 25656 (time 1767840655.68 total 47.51 last 565.58) - open/close 30000 (time 1767840665.27 total 57.09 last 453.36) - open/close 34505 (time 1767840675.27 total 67.09 last 450.49) - open/close 39472 (time 1767840685.27 total 77.09 last 496.66) - open/close 44270 (time 1767840695.27 total 87.10 last 479.74) - open/close 48835 (time 1767840705.27 total 97.10 last 456.43) - open/close 50000 (time 1767840707.88 total 99.71 last 445.65) - open/close 54509 (time 1767840717.89 total 109.71 last 450.84) - open/close 58896 (time 1767840727.89 total 119.71 last 438.62) - open/close 60000 (time 1767840730.29 total 122.12 last 459.14) - open/close 64176 (time 1767840740.29 total 132.12 last 417.59) total: 64768 open/close in 133.82 seconds: 484.01 ops/second - /unlink 8325 (time 1767840753.40 total 10.00 last 832.42) - /unlink 16538 (time 1767840763.40 total 20.00 last 821.13) - /unlink 20000 (time 1767840767.08 total 23.69 last 939.64) - /unlink 27743 (time 1767840777.09 total 33.69 last 774.28) - /unlink 30000 (time 1767840780.17 total 36.77 last 731.88) - /unlink 37903 (time 1767840790.17 total 46.77 last 790.28) - /unlink 40000 (time 1767840792.49 total 49.09 last 903.10) - /unlink 48565 (time 1767840802.49 total 59.09 last 856.44) - /unlink 50000 (time 1767840804.54 total 61.15 last 699.33) - /unlink 58888 (time 1767840814.55 total 71.15 last 888.70) total: 64768 /unlink in 77.75 seconds: 833.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5348 (time 1767840857.94 total 10.00 last 534.77) - open/close 10000 (time 1767840866.37 total 18.43 last 551.96) - open/close 15310 (time 1767840876.37 total 28.43 last 530.99) - open/close 20000 (time 1767840885.42 total 37.48 last 517.95) - open/close 25237 (time 1767840895.42 total 47.48 last 523.66) - open/close 30000 (time 1767840905.02 total 57.08 last 496.45) - open/close 35564 (time 1767840915.02 total 67.08 last 556.33) - open/close 40000 (time 1767840922.78 total 74.85 last 571.29) - open/close 44632 (time 1767840932.78 total 84.85 last 463.20) - open/close 48980 (time 1767840942.79 total 94.85 last 434.71) - open/close 50000 (time 1767840944.98 total 97.05 last 463.68) - open/close 55107 (time 1767840954.99 total 107.05 last 510.66) - open/close 59432 (time 1767840964.99 total 117.05 last 432.43) - open/close 61268 (time 1767840974.99 total 127.05 last 183.59) - open/close 62924 (time 1767840984.99 total 137.05 last 165.52) - open/close 63744 (time 1767840995.00 total 147.06 last 81.96) - open/close 64388 (time 1767841005.00 total 157.06 last 64.40) total: 64768 open/close in 161.29 seconds: 401.57 ops/second - /unlink 3598 (time 1767841022.74 total 10.00 last 359.78) - /unlink 7986 (time 1767841032.74 total 20.00 last 438.77) - /unlink 10000 (time 1767841038.37 total 25.63 last 357.92) - /unlink 13943 (time 1767841048.37 total 35.63 last 394.29) - /unlink 17243 (time 1767841058.37 total 45.63 last 329.90) - /unlink 20000 (time 1767841067.52 total 54.78 last 301.31) - /unlink 23182 (time 1767841077.52 total 64.78 last 318.15) - /unlink 27169 (time 1767841087.52 total 74.79 last 398.62) - /unlink 30000 (time 1767841093.59 total 80.85 last 466.77) - /unlink 35394 (time 1767841103.59 total 90.85 last 539.37) - /unlink 38794 (time 1767841113.59 total 100.85 last 339.93) - /unlink 40000 (time 1767841117.08 total 104.34 last 345.43) - /unlink 42581 (time 1767841127.08 total 114.35 last 258.08) - /unlink 45316 (time 1767841137.09 total 124.35 last 273.41) - /unlink 47808 (time 1767841147.09 total 134.35 last 249.11) - /unlink 50000 (time 1767841154.37 total 141.64 last 300.90) - /unlink 53011 (time 1767841164.38 total 151.64 last 300.94) - /unlink 57578 (time 1767841174.38 total 161.64 last 456.62) - /unlink 60000 (time 1767841181.90 total 169.17 last 321.98) total: 64768 /unlink in 177.11 seconds: 365.70 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3495 (time 1767841216.89 total 10.00 last 349.45) - open/close 6186 (time 1767841226.89 total 20.00 last 269.08) - open/close 8035 (time 1767841236.89 total 30.01 last 184.83) - open/close 10000 (time 1767841246.79 total 39.91 last 198.49) - open/close 12883 (time 1767841256.79 total 49.91 last 288.26) - open/close 15443 (time 1767841266.80 total 59.91 last 255.98) - open/close 18339 (time 1767841276.80 total 69.91 last 289.53) - open/close 20000 (time 1767841281.31 total 74.42 last 368.09) - open/close 24209 (time 1767841291.31 total 84.42 last 420.87) - open/close 28476 (time 1767841301.31 total 94.42 last 426.66) - open/close 30000 (time 1767841304.30 total 97.41 last 509.57) - open/close 34337 (time 1767841314.30 total 107.42 last 433.64) - open/close 38817 (time 1767841324.30 total 117.42 last 447.96) - open/close 40000 (time 1767841326.69 total 119.80 last 496.44) - open/close 45041 (time 1767841336.69 total 129.80 last 504.06) - open/close 50000 (time 1767841346.06 total 139.17 last 529.14) - open/close 55118 (time 1767841356.06 total 149.17 last 511.77) - open/close 59999 (time 1767841366.06 total 159.17 last 488.10) total: 64768 open/close in 169.15 seconds: 382.91 ops/second - /unlink 7314 (time 1767841387.42 total 10.00 last 731.35) - /unlink 10000 (time 1767841391.61 total 14.19 last 641.32) - /unlink 17420 (time 1767841401.61 total 24.19 last 742.00) - /unlink 20000 (time 1767841405.33 total 27.91 last 694.23) - /unlink 27595 (time 1767841415.33 total 37.91 last 759.43) - /unlink 30000 (time 1767841417.96 total 40.54 last 914.43) - /unlink 37345 (time 1767841427.96 total 50.54 last 734.43) - /unlink 40000 (time 1767841431.49 total 54.07 last 751.53) - /unlink 47623 (time 1767841441.49 total 64.07 last 762.20) - /unlink 50000 (time 1767841444.82 total 67.40 last 713.89) - /unlink 57641 (time 1767841454.83 total 77.41 last 763.77) - /unlink 60000 (time 1767841457.59 total 80.17 last 854.41) total: 64768 /unlink in 85.91 seconds: 753.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg427-server: LNET ready to unload oleg427-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2