== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:26:00 (1768008360) Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:-f) oleg325-server: oleg325-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg325-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg325-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1240 (time 1768008423.23 total 10.01 last 123.82) - open/close 2586 (time 1768008433.23 total 20.02 last 134.54) - open/close 3978 (time 1768008443.24 total 30.03 last 139.11) - open/close 5132 (time 1768008453.26 total 40.04 last 115.19) - open/close 6292 (time 1768008463.26 total 50.05 last 115.98) - open/close 7504 (time 1768008473.27 total 60.05 last 121.13) - open/close 8789 (time 1768008483.27 total 70.05 last 128.47) - open/close 10000 (time 1768008493.24 total 80.03 last 121.44) - open/close 11160 (time 1768008503.25 total 90.03 last 115.93) - open/close 12364 (time 1768008513.25 total 100.03 last 120.38) - open/close 13709 (time 1768008523.26 total 110.04 last 134.37) - open/close 14943 (time 1768008533.26 total 120.05 last 123.33) - open/close 16258 (time 1768008543.27 total 130.05 last 131.43) - open/close 17723 (time 1768008553.27 total 140.05 last 146.49) - open/close 19173 (time 1768008563.27 total 150.06 last 144.93) - open/close 20000 (time 1768008568.90 total 155.69 last 146.89) - open/close 21415 (time 1768008578.91 total 165.70 last 141.38) - open/close 22817 (time 1768008588.92 total 175.70 last 140.14) - open/close 24120 (time 1768008598.92 total 185.71 last 130.25) - open/close 25451 (time 1768008608.93 total 195.71 last 133.04) - open/close 26732 (time 1768008618.93 total 205.72 last 128.05) - open/close 28093 (time 1768008628.93 total 215.72 last 136.04) - open/close 29395 (time 1768008638.94 total 225.73 last 130.11) - open/close 30000 (time 1768008643.37 total 230.16 last 136.50) - open/close 31328 (time 1768008653.38 total 240.16 last 132.76) - open/close 32766 (time 1768008663.38 total 250.16 last 143.77) - open/close 34009 (time 1768008673.38 total 260.16 last 124.30) - open/close 35455 (time 1768008683.38 total 270.17 last 144.53) - open/close 37066 (time 1768008693.38 total 280.17 last 161.08) - open/close 38750 (time 1768008703.39 total 290.18 last 168.26) - open/close 40000 (time 1768008711.34 total 298.13 last 157.19) - open/close 42015 (time 1768008721.35 total 308.13 last 201.46) - open/close 43723 (time 1768008731.35 total 318.14 last 170.74) - open/close 46134 (time 1768008741.35 total 328.14 last 241.03) - open/close 48379 (time 1768008751.35 total 338.14 last 224.49) - open/close 50000 (time 1768008759.33 total 346.12 last 203.19) - open/close 51962 (time 1768008769.33 total 356.12 last 196.17) - open/close 53883 (time 1768008779.34 total 366.12 last 192.00) - open/close 56080 (time 1768008789.34 total 376.13 last 219.64) - open/close 58863 (time 1768008799.34 total 386.13 last 278.26) - open/close 60000 (time 1768008802.62 total 389.41 last 346.96) - open/close 62894 (time 1768008812.62 total 399.41 last 289.32) total: 64768 open/close in 406.66 seconds: 159.27 ops/second - /unlink 5405 (time 1768008831.36 total 10.00 last 540.46) - /unlink 10000 (time 1768008840.07 total 18.72 last 527.24) - /unlink 14540 (time 1768008850.08 total 28.72 last 453.84) - /unlink 19737 (time 1768008860.08 total 38.72 last 519.69) - /unlink 22978 (time 1768008870.08 total 48.72 last 324.08) - /unlink 27586 (time 1768008880.08 total 58.72 last 460.66) - /unlink 30000 (time 1768008886.02 total 64.67 last 406.26) - /unlink 34440 (time 1768008896.02 total 74.67 last 443.99) - /unlink 38150 (time 1768008906.02 total 84.67 last 370.97) - /unlink 40000 (time 1768008910.79 total 89.43 last 388.55) - /unlink 44203 (time 1768008920.79 total 99.43 last 420.16) - /unlink 49137 (time 1768008930.79 total 109.43 last 493.36) - /unlink 53893 (time 1768008940.79 total 119.43 last 475.52) - /unlink 58352 (time 1768008950.80 total 129.44 last 445.70) - /unlink 60000 (time 1768008954.82 total 133.47 last 409.10) - /unlink 64201 (time 1768008964.83 total 143.47 last 420.05) total: 64768 /unlink in 144.73 seconds: 447.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3044 (time 1768008997.58 total 10.00 last 304.33) - open/close 6126 (time 1768009007.58 total 20.00 last 308.17) - open/close 8998 (time 1768009017.59 total 30.01 last 287.10) - open/close 10000 (time 1768009020.88 total 33.31 last 303.79) - open/close 12265 (time 1768009030.89 total 43.31 last 226.46) - open/close 15262 (time 1768009040.89 total 53.31 last 299.67) - open/close 18373 (time 1768009050.89 total 63.31 last 311.03) - open/close 20000 (time 1768009056.75 total 69.18 last 277.40) - open/close 22995 (time 1768009066.75 total 79.18 last 299.47) - open/close 25808 (time 1768009076.76 total 89.18 last 281.24) - open/close 28347 (time 1768009086.76 total 99.18 last 253.83) - open/close 30000 (time 1768009092.53 total 104.95 last 286.57) - open/close 32936 (time 1768009102.53 total 114.95 last 293.50) - open/close 35549 (time 1768009112.54 total 124.96 last 261.20) - open/close 38013 (time 1768009122.54 total 134.96 last 246.30) - open/close 40000 (time 1768009130.19 total 142.61 last 259.67) - open/close 43544 (time 1768009140.19 total 152.61 last 354.37) - open/close 46587 (time 1768009150.20 total 162.62 last 304.19) - open/close 49875 (time 1768009160.20 total 172.62 last 328.73) - open/close 52712 (time 1768009170.20 total 182.62 last 283.70) - open/close 55657 (time 1768009180.20 total 192.62 last 294.42) - open/close 58991 (time 1768009190.20 total 202.62 last 333.35) - open/close 60000 (time 1768009192.98 total 205.40 last 363.22) - open/close 63479 (time 1768009202.98 total 215.40 last 347.86) total: 64768 open/close in 218.66 seconds: 296.21 ops/second - /unlink 5894 (time 1768009217.71 total 10.00 last 589.36) - /unlink 10000 (time 1768009223.84 total 16.13 last 669.85) - /unlink 16631 (time 1768009233.84 total 26.13 last 663.09) - /unlink 20000 (time 1768009239.09 total 31.38 last 642.25) - /unlink 25567 (time 1768009249.09 total 41.38 last 556.64) - /unlink 30000 (time 1768009255.86 total 48.15 last 654.75) - /unlink 36336 (time 1768009265.86 total 58.15 last 633.57) - /unlink 40000 (time 1768009271.16 total 63.44 last 691.93) - /unlink 46215 (time 1768009281.16 total 73.44 last 621.46) - /unlink 50000 (time 1768009287.34 total 79.63 last 611.99) - /unlink 56637 (time 1768009297.35 total 89.63 last 663.42) - /unlink 60000 (time 1768009302.10 total 94.38 last 708.26) total: 64768 /unlink in 101.35 seconds: 639.05 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4627 (time 1768009363.55 total 10.00 last 462.65) - open/close 9396 (time 1768009373.55 total 20.00 last 476.90) - open/close 13329 (time 1768009383.56 total 30.01 last 393.07) - open/close 17178 (time 1768009393.56 total 40.01 last 384.81) - open/close 20000 (time 1768009401.79 total 48.24 last 342.90) - open/close 24026 (time 1768009411.79 total 58.24 last 402.60) - open/close 27439 (time 1768009421.79 total 68.24 last 341.29) - open/close 30000 (time 1768009428.34 total 74.79 last 390.89) - open/close 33954 (time 1768009438.34 total 84.79 last 395.36) - open/close 37827 (time 1768009448.34 total 94.79 last 387.25) - open/close 40000 (time 1768009453.31 total 99.76 last 437.73) - open/close 43716 (time 1768009463.31 total 109.76 last 371.54) - open/close 47891 (time 1768009473.31 total 119.76 last 417.49) - open/close 50000 (time 1768009479.33 total 125.78 last 350.19) - open/close 54342 (time 1768009489.33 total 135.78 last 434.18) - open/close 58527 (time 1768009499.33 total 145.78 last 418.41) - open/close 60000 (time 1768009502.66 total 149.11 last 443.13) total: 64768 open/close in 158.46 seconds: 408.72 ops/second - /unlink 7404 (time 1768009523.42 total 10.00 last 740.37) - /unlink 10000 (time 1768009527.81 total 14.39 last 591.56) - /unlink 17165 (time 1768009537.81 total 24.39 last 716.44) - /unlink 20000 (time 1768009541.80 total 28.38 last 710.57) - /unlink 26152 (time 1768009551.80 total 38.38 last 615.05) - /unlink 30000 (time 1768009557.24 total 43.82 last 707.32) - /unlink 36695 (time 1768009567.24 total 53.82 last 669.45) - /unlink 40000 (time 1768009572.19 total 58.77 last 667.58) - /unlink 45063 (time 1768009582.19 total 68.78 last 506.23) - /unlink 50000 (time 1768009591.52 total 78.11 last 529.12) - /unlink 56338 (time 1768009601.52 total 88.11 last 633.74) - /unlink 60000 (time 1768009607.40 total 93.99 last 622.87) total: 64768 /unlink in 103.42 seconds: 626.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4346 (time 1768009647.18 total 10.00 last 434.57) - open/close 8539 (time 1768009657.18 total 20.00 last 419.23) - open/close 10000 (time 1768009661.00 total 23.83 last 382.06) - open/close 14413 (time 1768009671.00 total 33.83 last 441.29) - open/close 19010 (time 1768009681.00 total 43.83 last 459.69) - open/close 20000 (time 1768009683.13 total 45.96 last 464.90) - open/close 23607 (time 1768009693.13 total 55.96 last 360.63) - open/close 26892 (time 1768009703.13 total 65.96 last 328.47) - open/close 30000 (time 1768009711.79 total 74.61 last 359.14) - open/close 33169 (time 1768009721.79 total 84.61 last 316.89) - open/close 35759 (time 1768009731.79 total 94.62 last 258.95) - open/close 38695 (time 1768009741.79 total 104.62 last 293.51) - open/close 40000 (time 1768009746.48 total 109.31 last 278.25) - open/close 42796 (time 1768009756.48 total 119.31 last 279.57) - open/close 45460 (time 1768009766.49 total 129.31 last 266.28) - open/close 48340 (time 1768009776.49 total 139.32 last 287.97) - open/close 50000 (time 1768009783.41 total 146.23 last 239.96) - open/close 52576 (time 1768009793.41 total 156.23 last 257.60) - open/close 55873 (time 1768009803.41 total 166.24 last 329.55) - open/close 59529 (time 1768009813.41 total 176.24 last 365.56) - open/close 63101 (time 1768009823.42 total 186.24 last 357.16) total: 64768 open/close in 190.04 seconds: 340.82 ops/second - /unlink 6414 (time 1768009838.54 total 10.00 last 641.39) - /unlink 10000 (time 1768009843.53 total 15.00 last 717.89) - /unlink 16609 (time 1768009853.53 total 25.00 last 660.84) - /unlink 20000 (time 1768009858.23 total 29.69 last 721.81) - /unlink 27073 (time 1768009868.23 total 39.69 last 707.27) - /unlink 30000 (time 1768009872.43 total 43.90 last 696.76) - /unlink 37182 (time 1768009882.43 total 53.90 last 718.17) - /unlink 40000 (time 1768009886.28 total 57.74 last 733.18) - /unlink 47766 (time 1768009896.28 total 67.74 last 776.53) - /unlink 50000 (time 1768009900.08 total 71.55 last 587.14) - /unlink 56318 (time 1768009910.08 total 81.55 last 631.75) - /unlink 60000 (time 1768009914.96 total 86.42 last 754.77) total: 64768 /unlink in 93.57 seconds: 692.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3755 (time 1768010013.65 total 10.00 last 375.39) - open/close 8271 (time 1768010023.65 total 20.00 last 451.58) - open/close 10000 (time 1768010027.51 total 23.86 last 448.37) - open/close 13640 (time 1768010037.51 total 33.87 last 363.75) - open/close 17125 (time 1768010047.51 total 43.87 last 348.44) - open/close 20000 (time 1768010055.74 total 52.09 last 349.69) - open/close 24114 (time 1768010065.74 total 62.09 last 411.37) - open/close 28755 (time 1768010075.74 total 72.09 last 464.04) - open/close 30000 (time 1768010078.58 total 74.93 last 438.10) - open/close 34226 (time 1768010088.58 total 84.93 last 422.54) - open/close 39108 (time 1768010098.58 total 94.94 last 488.14) - open/close 43404 (time 1768010108.58 total 104.94 last 429.55) - open/close 47001 (time 1768010118.59 total 114.94 last 359.65) - open/close 50000 (time 1768010126.07 total 122.42 last 400.92) - open/close 53311 (time 1768010136.07 total 132.42 last 331.07) - open/close 57393 (time 1768010146.07 total 142.42 last 408.12) - open/close 60000 (time 1768010152.27 total 148.62 last 420.33) - open/close 64319 (time 1768010162.27 total 158.63 last 431.83) total: 64768 open/close in 159.73 seconds: 405.49 ops/second - /unlink 7284 (time 1768010174.75 total 10.00 last 728.36) - /unlink 10000 (time 1768010177.95 total 13.21 last 846.72) - /unlink 17539 (time 1768010187.96 total 23.21 last 753.83) - /unlink 20000 (time 1768010192.04 total 27.29 last 602.93) - /unlink 26981 (time 1768010202.04 total 37.29 last 698.04) - /unlink 30000 (time 1768010206.00 total 41.25 last 761.99) - /unlink 37752 (time 1768010216.00 total 51.25 last 775.14) - /unlink 40000 (time 1768010218.67 total 53.92 last 841.96) - /unlink 46191 (time 1768010228.67 total 63.92 last 619.09) - /unlink 50000 (time 1768010234.78 total 70.04 last 623.03) - /unlink 56114 (time 1768010244.80 total 80.05 last 610.46) - /unlink 60000 (time 1768010251.36 total 86.62 last 591.88) total: 64768 /unlink in 93.13 seconds: 695.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local oleg325-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2