== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:10:17 (1758823817) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) oleg151-server: oleg151-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg151-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg151-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1885 (time 1758823863.25 total 10.00 last 188.46) - open/close 3260 (time 1758823873.26 total 20.01 last 137.38) - open/close 4543 (time 1758823883.26 total 30.01 last 128.26) - open/close 5595 (time 1758823893.26 total 40.02 last 105.16) - open/close 6336 (time 1758823903.29 total 50.04 last 73.91) - open/close 7155 (time 1758823913.30 total 60.05 last 81.81) - open/close 8007 (time 1758823923.30 total 70.05 last 85.20) - open/close 9200 (time 1758823933.32 total 80.07 last 119.07) - open/close 10000 (time 1758823940.07 total 86.82 last 118.52) - open/close 11438 (time 1758823950.08 total 96.83 last 143.69) - open/close 13011 (time 1758823960.08 total 106.83 last 157.25) - open/close 14427 (time 1758823970.09 total 116.84 last 141.48) - open/close 15886 (time 1758823980.09 total 126.85 last 145.86) - open/close 17887 (time 1758823990.10 total 136.85 last 200.00) - open/close 19711 (time 1758824000.10 total 146.85 last 182.37) - open/close 22405 (time 1758824010.13 total 156.89 last 268.50) - open/close 24195 (time 1758824020.14 total 166.89 last 178.94) - open/close 25775 (time 1758824030.14 total 176.90 last 157.90) - open/close 27147 (time 1758824040.15 total 186.90 last 137.15) - open/close 28544 (time 1758824050.15 total 196.90 last 139.66) - open/close 30000 (time 1758824059.65 total 206.41 last 153.15) - open/close 31897 (time 1758824069.66 total 216.41 last 189.61) - open/close 33553 (time 1758824079.66 total 226.42 last 165.55) - open/close 34903 (time 1758824089.67 total 236.42 last 134.96) - open/close 36144 (time 1758824099.67 total 246.42 last 124.05) - open/close 37324 (time 1758824109.67 total 256.43 last 117.94) - open/close 38485 (time 1758824119.68 total 266.43 last 116.08) - open/close 39549 (time 1758824129.68 total 276.43 last 106.38) - open/close 40000 (time 1758824132.83 total 279.58 last 143.12) - open/close 41329 (time 1758824142.83 total 289.58 last 132.89) - open/close 42581 (time 1758824152.83 total 299.59 last 125.19) - open/close 43890 (time 1758824162.83 total 309.59 last 130.85) - open/close 45041 (time 1758824172.84 total 319.59 last 115.08) - open/close 46147 (time 1758824182.84 total 329.59 last 110.59) - open/close 47198 (time 1758824192.85 total 339.60 last 105.01) - open/close 48227 (time 1758824202.85 total 349.60 last 102.89) - open/close 49256 (time 1758824212.85 total 359.60 last 102.88) - open/close 50000 (time 1758824221.09 total 367.85 last 90.23) - open/close 51090 (time 1758824231.10 total 377.86 last 108.92) - open/close 52096 (time 1758824241.11 total 387.87 last 100.49) - open/close 53292 (time 1758824251.12 total 397.87 last 119.52) - open/close 54758 (time 1758824261.12 total 407.88 last 146.53) - open/close 56077 (time 1758824271.13 total 417.88 last 131.83) - open/close 57988 (time 1758824281.13 total 427.89 last 191.00) - open/close 59663 (time 1758824291.13 total 437.89 last 167.49) - open/close 61960 (time 1758824301.14 total 447.89 last 229.69) - open/close 63527 (time 1758824311.14 total 457.89 last 156.65) total: 64768 open/close in 463.88 seconds: 139.62 ops/second - /unlink 3867 (time 1758824328.94 total 10.00 last 386.62) - /unlink 7850 (time 1758824338.94 total 20.00 last 398.30) - /unlink 10000 (time 1758824343.71 total 24.77 last 450.55) - /unlink 13553 (time 1758824353.71 total 34.78 last 355.23) - /unlink 16821 (time 1758824363.71 total 44.78 last 326.76) - /unlink 20000 (time 1758824372.95 total 54.01 last 344.23) - /unlink 23519 (time 1758824382.95 total 64.01 last 351.87) - /unlink 26739 (time 1758824392.95 total 74.01 last 321.97) - /unlink 29820 (time 1758824402.95 total 84.02 last 308.06) - /unlink 33956 (time 1758824412.95 total 94.02 last 413.52) - /unlink 37764 (time 1758824422.95 total 104.02 last 380.76) - /unlink 40000 (time 1758824427.65 total 108.72 last 475.91) - /unlink 44557 (time 1758824437.65 total 118.72 last 455.63) - /unlink 47937 (time 1758824447.65 total 128.72 last 337.97) - /unlink 50000 (time 1758824452.87 total 133.94 last 395.30) - /unlink 53870 (time 1758824462.87 total 143.94 last 386.96) - /unlink 57532 (time 1758824472.87 total 153.94 last 366.20) - /unlink 60000 (time 1758824478.53 total 159.59 last 436.41) - /unlink 63999 (time 1758824488.53 total 169.59 last 399.90) total: 64768 /unlink in 171.71 seconds: 377.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2388 (time 1758824524.39 total 10.00 last 238.80) - open/close 4476 (time 1758824534.39 total 20.00 last 208.78) - open/close 6247 (time 1758824544.39 total 30.00 last 177.06) - open/close 7952 (time 1758824554.40 total 40.02 last 170.29) - open/close 10000 (time 1758824564.03 total 49.64 last 212.70) - open/close 11799 (time 1758824574.04 total 59.65 last 179.86) - open/close 14011 (time 1758824584.04 total 69.65 last 221.18) - open/close 16112 (time 1758824594.04 total 79.65 last 210.05) - open/close 18364 (time 1758824604.04 total 89.65 last 225.19) - open/close 20000 (time 1758824611.45 total 97.06 last 220.81) - open/close 21536 (time 1758824621.45 total 107.06 last 153.55) - open/close 23340 (time 1758824631.46 total 117.07 last 180.32) - open/close 25108 (time 1758824641.46 total 127.07 last 176.67) - open/close 27153 (time 1758824651.47 total 137.08 last 204.43) - open/close 29640 (time 1758824661.47 total 147.08 last 248.63) - open/close 31913 (time 1758824671.47 total 157.08 last 227.26) - open/close 34023 (time 1758824681.47 total 167.08 last 210.98) - open/close 36536 (time 1758824691.48 total 177.09 last 251.21) - open/close 39122 (time 1758824701.48 total 187.09 last 258.58) - open/close 40000 (time 1758824705.49 total 191.10 last 218.62) - open/close 42072 (time 1758824715.50 total 201.11 last 207.11) - open/close 44146 (time 1758824725.50 total 211.11 last 207.34) - open/close 46118 (time 1758824735.50 total 221.11 last 197.18) - open/close 48482 (time 1758824745.50 total 231.11 last 236.36) - open/close 50000 (time 1758824751.18 total 236.79 last 267.31) - open/close 51857 (time 1758824761.18 total 246.79 last 185.66) - open/close 54072 (time 1758824771.19 total 256.80 last 221.44) - open/close 55841 (time 1758824781.19 total 266.80 last 176.90) - open/close 57960 (time 1758824791.19 total 276.80 last 211.85) - open/close 60000 (time 1758824800.90 total 286.51 last 210.15) - open/close 62513 (time 1758824810.90 total 296.51 last 251.22) total: 64768 open/close in 305.79 seconds: 211.81 ops/second - /unlink 4603 (time 1758824831.80 total 10.00 last 460.27) - /unlink 9761 (time 1758824841.80 total 20.00 last 515.77) - /unlink 14431 (time 1758824851.80 total 30.00 last 466.96) - /unlink 18606 (time 1758824861.80 total 40.00 last 417.49) - /unlink 20000 (time 1758824864.43 total 42.64 last 529.15) - /unlink 24734 (time 1758824874.44 total 52.64 last 473.34) - /unlink 29408 (time 1758824884.44 total 62.64 last 467.22) - /unlink 33695 (time 1758824894.44 total 72.65 last 428.56) - /unlink 38197 (time 1758824904.44 total 82.65 last 450.13) - /unlink 40000 (time 1758824908.11 total 86.31 last 492.16) - /unlink 43865 (time 1758824918.11 total 96.31 last 386.42) - /unlink 47830 (time 1758824928.12 total 106.32 last 396.29) - /unlink 50000 (time 1758824932.84 total 111.05 last 458.91) - /unlink 55227 (time 1758824942.84 total 121.05 last 522.67) - /unlink 60000 (time 1758824951.42 total 129.62 last 556.57) total: 64768 /unlink in 137.04 seconds: 472.62 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3367 (time 1758824991.50 total 10.00 last 336.60) - open/close 6788 (time 1758825001.50 total 20.01 last 342.03) - open/close 9078 (time 1758825011.50 total 30.01 last 228.89) - open/close 10000 (time 1758825015.51 total 34.01 last 230.48) - open/close 13218 (time 1758825025.51 total 44.01 last 321.78) - open/close 15972 (time 1758825035.51 total 54.01 last 275.34) - open/close 18805 (time 1758825045.51 total 64.02 last 283.20) - open/close 20000 (time 1758825048.77 total 67.28 last 366.20) - open/close 23572 (time 1758825058.78 total 77.28 last 357.17) - open/close 28038 (time 1758825068.78 total 87.28 last 446.59) - open/close 30000 (time 1758825073.22 total 91.73 last 441.44) - open/close 34134 (time 1758825083.22 total 101.73 last 413.35) - open/close 38709 (time 1758825093.22 total 111.73 last 457.45) - open/close 40000 (time 1758825096.35 total 114.86 last 412.67) - open/close 44223 (time 1758825106.35 total 124.86 last 422.30) - open/close 48361 (time 1758825116.35 total 134.86 last 413.74) - open/close 50000 (time 1758825120.20 total 138.71 last 425.97) - open/close 53797 (time 1758825130.20 total 148.71 last 379.68) - open/close 57436 (time 1758825140.20 total 158.71 last 363.82) - open/close 60000 (time 1758825146.92 total 165.42 last 381.93) - open/close 63263 (time 1758825156.92 total 175.42 last 326.30) total: 64768 open/close in 179.41 seconds: 361.00 ops/second - /unlink 5308 (time 1758825172.36 total 10.00 last 530.73) - /unlink 10000 (time 1758825180.22 total 17.86 last 596.87) - /unlink 16982 (time 1758825190.22 total 27.86 last 698.15) - /unlink 20000 (time 1758825194.83 total 32.47 last 654.59) - /unlink 26547 (time 1758825204.83 total 42.47 last 654.64) - /unlink 30000 (time 1758825211.77 total 49.41 last 498.07) - /unlink 34713 (time 1758825221.77 total 59.41 last 471.25) - /unlink 40000 (time 1758825230.33 total 67.97 last 617.65) - /unlink 45669 (time 1758825240.33 total 77.97 last 566.88) - /unlink 50000 (time 1758825246.76 total 84.40 last 673.43) - /unlink 56151 (time 1758825256.76 total 94.40 last 615.10) - /unlink 60000 (time 1758825263.30 total 100.94 last 588.18) - /unlink 64434 (time 1758825273.30 total 110.94 last 443.35) total: 64768 /unlink in 111.53 seconds: 580.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2903 (time 1758825301.97 total 10.00 last 290.23) - open/close 5737 (time 1758825311.97 total 20.00 last 283.38) - open/close 9027 (time 1758825321.97 total 30.01 last 328.90) - open/close 10000 (time 1758825324.33 total 32.36 last 412.59) - open/close 14245 (time 1758825334.33 total 42.37 last 424.39) - open/close 18218 (time 1758825344.33 total 52.37 last 397.26) - open/close 20000 (time 1758825349.11 total 57.15 last 372.78) - open/close 23492 (time 1758825359.12 total 67.15 last 349.17) - open/close 26911 (time 1758825369.12 total 77.15 last 341.86) - open/close 30000 (time 1758825377.78 total 85.82 last 356.46) - open/close 33752 (time 1758825387.78 total 95.82 last 375.14) - open/close 37817 (time 1758825397.79 total 105.82 last 406.46) - open/close 40000 (time 1758825402.45 total 110.48 last 468.28) - open/close 43724 (time 1758825412.45 total 120.48 last 372.35) - open/close 47680 (time 1758825422.45 total 130.48 last 395.56) - open/close 50000 (time 1758825428.65 total 136.68 last 374.20) - open/close 53991 (time 1758825438.65 total 146.68 last 399.02) - open/close 58139 (time 1758825448.65 total 156.69 last 414.78) - open/close 60000 (time 1758825453.23 total 161.26 last 406.76) - open/close 63908 (time 1758825463.23 total 171.26 last 390.74) total: 64768 open/close in 173.41 seconds: 373.51 ops/second - /unlink 7088 (time 1758825476.83 total 10.00 last 708.75) - /unlink 10000 (time 1758825480.51 total 13.69 last 789.97) - /unlink 17470 (time 1758825490.51 total 23.69 last 746.97) - /unlink 20000 (time 1758825494.34 total 27.52 last 660.54) - /unlink 26961 (time 1758825504.34 total 37.52 last 696.02) - /unlink 30000 (time 1758825508.66 total 41.83 last 704.61) - /unlink 37510 (time 1758825518.66 total 51.83 last 750.94) - /unlink 40000 (time 1758825521.41 total 54.59 last 904.10) - /unlink 47216 (time 1758825531.41 total 64.59 last 721.54) - /unlink 50000 (time 1758825535.41 total 68.59 last 696.11) - /unlink 57129 (time 1758825545.41 total 78.59 last 712.84) - /unlink 60000 (time 1758825549.24 total 82.42 last 749.28) total: 64768 /unlink in 88.84 seconds: 729.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4414 (time 1758825586.33 total 10.00 last 441.40) - open/close 9702 (time 1758825596.33 total 20.00 last 528.72) - open/close 14941 (time 1758825606.33 total 30.00 last 523.86) - open/close 20000 (time 1758825616.01 total 39.68 last 522.93) - open/close 24243 (time 1758825626.01 total 49.68 last 424.24) - open/close 28577 (time 1758825636.01 total 59.68 last 433.28) - open/close 30000 (time 1758825639.22 total 62.89 last 443.15) - open/close 34280 (time 1758825649.22 total 72.89 last 428.00) - open/close 38349 (time 1758825659.22 total 82.89 last 406.88) - open/close 40000 (time 1758825663.57 total 87.24 last 379.86) - open/close 43903 (time 1758825673.57 total 97.24 last 390.25) - open/close 47899 (time 1758825683.57 total 107.24 last 399.59) - open/close 50000 (time 1758825688.59 total 112.26 last 418.83) - open/close 54235 (time 1758825698.59 total 122.26 last 423.39) - open/close 58563 (time 1758825708.59 total 132.26 last 432.79) - open/close 60000 (time 1758825712.14 total 135.81 last 404.60) - open/close 64709 (time 1758825722.15 total 145.82 last 470.43) total: 64768 open/close in 145.95 seconds: 443.77 ops/second - /unlink 7267 (time 1758825733.64 total 10.00 last 726.63) - /unlink 10000 (time 1758825738.39 total 14.75 last 575.05) - /unlink 15146 (time 1758825748.39 total 24.76 last 514.43) - /unlink 20000 (time 1758825756.54 total 32.90 last 595.76) - /unlink 26707 (time 1758825766.54 total 42.91 last 670.60) - /unlink 30000 (time 1758825771.36 total 47.73 last 682.82) - /unlink 36603 (time 1758825781.36 total 57.73 last 660.27) - /unlink 40000 (time 1758825786.29 total 62.65 last 690.37) - /unlink 46907 (time 1758825796.29 total 72.65 last 690.64) - /unlink 50000 (time 1758825800.49 total 76.86 last 735.35) - /unlink 57004 (time 1758825810.49 total 86.86 last 700.32) - /unlink 60000 (time 1758825814.20 total 90.56 last 808.59) total: 64768 /unlink in 97.40 seconds: 664.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg151-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2