== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:50:53 (1754009453) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg152-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1023 (time 1754009530.89 total 10.01 last 102.21) - open/close 2016 (time 1754009540.89 total 20.01 last 99.29) - open/close 3292 (time 1754009550.89 total 30.02 last 127.54) - open/close 4477 (time 1754009560.89 total 40.02 last 118.49) - open/close 5731 (time 1754009570.90 total 50.02 last 125.35) - open/close 7030 (time 1754009580.90 total 60.02 last 129.87) - open/close 8701 (time 1754009590.90 total 70.02 last 167.08) - open/close 10000 (time 1754009598.99 total 78.11 last 160.65) - open/close 11910 (time 1754009608.99 total 88.11 last 190.96) - open/close 13752 (time 1754009618.99 total 98.11 last 184.14) - open/close 15500 (time 1754009628.99 total 108.12 last 174.78) - open/close 17797 (time 1754009639.00 total 118.12 last 229.64) - open/close 20000 (time 1754009648.02 total 127.14 last 244.07) - open/close 22608 (time 1754009658.02 total 137.15 last 260.77) - open/close 24830 (time 1754009668.02 total 147.15 last 222.17) - open/close 27464 (time 1754009678.03 total 157.15 last 263.37) - open/close 29889 (time 1754009688.03 total 167.15 last 242.38) - open/close 32226 (time 1754009698.03 total 177.15 last 233.66) - open/close 34573 (time 1754009708.03 total 187.16 last 234.67) - open/close 37792 (time 1754009718.04 total 197.16 last 321.80) - open/close 40000 (time 1754009727.41 total 206.53 last 235.51) - open/close 42177 (time 1754009737.42 total 216.54 last 217.57) - open/close 44083 (time 1754009747.42 total 226.54 last 190.57) - open/close 46281 (time 1754009757.42 total 236.54 last 219.79) - open/close 48728 (time 1754009767.42 total 246.54 last 244.64) - open/close 50000 (time 1754009773.20 total 252.33 last 220.03) - open/close 52615 (time 1754009783.20 total 262.33 last 261.47) - open/close 55455 (time 1754009793.21 total 272.33 last 283.98) - open/close 58979 (time 1754009803.21 total 282.33 last 352.32) - open/close 60000 (time 1754009805.78 total 284.90 last 397.67) - open/close 62950 (time 1754009815.78 total 294.90 last 294.99) total: 64768 open/close in 300.96 seconds: 215.21 ops/second - /unlink 5814 (time 1754009833.24 total 10.00 last 581.34) - /unlink 10000 (time 1754009839.79 total 16.55 last 638.84) - /unlink 16302 (time 1754009849.79 total 26.55 last 630.14) - /unlink 20000 (time 1754009856.21 total 32.97 last 576.43) - /unlink 26027 (time 1754009866.21 total 42.97 last 602.62) - /unlink 30000 (time 1754009872.96 total 49.73 last 588.23) - /unlink 35371 (time 1754009882.97 total 59.73 last 537.04) - /unlink 40000 (time 1754009889.77 total 66.53 last 679.99) - /unlink 46494 (time 1754009899.77 total 76.53 last 649.36) - /unlink 50000 (time 1754009905.11 total 81.87 last 657.50) - /unlink 57656 (time 1754009915.11 total 91.87 last 765.56) - /unlink 60000 (time 1754009918.09 total 94.85 last 784.84) total: 64768 /unlink in 101.38 seconds: 638.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4337 (time 1754009954.86 total 10.00 last 433.63) - open/close 8504 (time 1754009964.86 total 20.00 last 416.67) - open/close 10000 (time 1754009967.98 total 23.12 last 479.70) - open/close 14830 (time 1754009977.98 total 33.12 last 482.98) - open/close 18964 (time 1754009987.98 total 43.12 last 413.30) - open/close 20000 (time 1754009990.43 total 45.58 last 422.31) - open/close 23872 (time 1754010000.43 total 55.58 last 387.18) - open/close 28271 (time 1754010010.43 total 65.58 last 439.87) - open/close 30000 (time 1754010014.45 total 69.59 last 431.03) - open/close 34578 (time 1754010024.45 total 79.59 last 457.79) - open/close 39612 (time 1754010034.45 total 89.59 last 503.36) - open/close 44523 (time 1754010044.45 total 99.59 last 491.01) - open/close 49340 (time 1754010054.45 total 109.59 last 481.68) - open/close 53987 (time 1754010064.45 total 119.59 last 464.62) - open/close 58813 (time 1754010074.45 total 129.59 last 482.60) - open/close 60000 (time 1754010076.95 total 132.10 last 474.14) total: 64768 open/close in 141.92 seconds: 456.37 ops/second - /unlink 7821 (time 1754010098.05 total 10.00 last 782.06) - /unlink 10000 (time 1754010100.42 total 12.38 last 916.32) - /unlink 18168 (time 1754010110.42 total 22.38 last 816.73) - /unlink 20000 (time 1754010112.92 total 24.87 last 734.79) - /unlink 28537 (time 1754010122.92 total 34.87 last 853.63) - /unlink 36864 (time 1754010132.92 total 44.87 last 832.65) - /unlink 40000 (time 1754010137.35 total 49.31 last 707.70) - /unlink 47839 (time 1754010147.35 total 59.31 last 783.85) - /unlink 50000 (time 1754010149.90 total 61.85 last 848.46) - /unlink 57512 (time 1754010159.90 total 71.85 last 751.18) - /unlink 60000 (time 1754010163.19 total 75.15 last 755.25) total: 64768 /unlink in 81.60 seconds: 793.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4168 (time 1754010234.31 total 10.00 last 416.75) - open/close 8490 (time 1754010244.32 total 20.00 last 432.06) - open/close 10000 (time 1754010247.32 total 23.00 last 503.38) - open/close 14856 (time 1754010257.32 total 33.00 last 485.57) - open/close 19937 (time 1754010267.32 total 43.01 last 508.07) - open/close 25092 (time 1754010277.32 total 53.01 last 515.47) - open/close 30000 (time 1754010286.75 total 62.44 last 520.08) - open/close 34504 (time 1754010296.76 total 72.44 last 450.34) - open/close 38923 (time 1754010306.76 total 82.44 last 441.87) - open/close 40000 (time 1754010309.10 total 84.79 last 459.72) - open/close 44457 (time 1754010319.10 total 94.79 last 445.64) - open/close 48695 (time 1754010329.13 total 104.81 last 422.75) - open/close 50000 (time 1754010332.37 total 108.06 last 402.37) - open/close 54160 (time 1754010342.37 total 118.06 last 415.98) - open/close 58158 (time 1754010352.37 total 128.06 last 399.75) - open/close 60000 (time 1754010356.95 total 132.63 last 402.53) - open/close 64164 (time 1754010366.95 total 142.64 last 416.31) total: 64768 open/close in 144.03 seconds: 449.68 ops/second - /unlink 7301 (time 1754010379.76 total 10.00 last 730.03) - /unlink 10000 (time 1754010383.60 total 13.84 last 703.61) - /unlink 16866 (time 1754010393.60 total 23.84 last 686.56) - /unlink 20000 (time 1754010398.01 total 28.25 last 709.83) - /unlink 27105 (time 1754010408.01 total 38.25 last 710.46) - /unlink 30000 (time 1754010411.69 total 41.93 last 787.33) - /unlink 37456 (time 1754010421.69 total 51.93 last 745.55) - /unlink 40000 (time 1754010425.19 total 55.43 last 727.85) - /unlink 47355 (time 1754010435.19 total 65.43 last 735.46) - /unlink 50000 (time 1754010439.05 total 69.29 last 685.37) - /unlink 57203 (time 1754010449.05 total 79.29 last 720.22) - /unlink 60000 (time 1754010452.82 total 83.06 last 741.75) total: 64768 /unlink in 89.59 seconds: 722.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4799 (time 1754010489.16 total 10.00 last 479.86) - open/close 9459 (time 1754010499.16 total 20.00 last 465.95) - open/close 14083 (time 1754010509.20 total 30.04 last 460.61) - open/close 18738 (time 1754010519.20 total 40.04 last 465.43) - open/close 20000 (time 1754010521.94 total 42.78 last 460.48) - open/close 24621 (time 1754010531.94 total 52.78 last 462.03) - open/close 29194 (time 1754010541.94 total 62.78 last 457.28) - open/close 33722 (time 1754010551.95 total 72.78 last 452.79) - open/close 38352 (time 1754010561.95 total 82.79 last 462.89) - open/close 40000 (time 1754010565.45 total 86.29 last 470.77) - open/close 44448 (time 1754010575.45 total 96.29 last 444.78) - open/close 48924 (time 1754010585.45 total 106.29 last 447.57) - open/close 50000 (time 1754010587.99 total 108.83 last 424.16) - open/close 54777 (time 1754010597.99 total 118.83 last 477.63) - open/close 59503 (time 1754010607.99 total 128.83 last 472.49) - open/close 63836 (time 1754010617.99 total 138.83 last 433.24) total: 64768 open/close in 140.97 seconds: 459.46 ops/second - /unlink 7610 (time 1754010631.49 total 10.00 last 760.95) - /unlink 10000 (time 1754010634.95 total 13.46 last 690.33) - /unlink 17293 (time 1754010644.95 total 23.46 last 729.18) - /unlink 20000 (time 1754010648.27 total 26.78 last 815.32) - /unlink 27292 (time 1754010658.27 total 36.79 last 729.16) - /unlink 30000 (time 1754010662.02 total 40.53 last 723.27) - /unlink 37281 (time 1754010672.02 total 50.53 last 728.04) - /unlink 40000 (time 1754010675.97 total 54.48 last 689.16) - /unlink 47498 (time 1754010685.97 total 64.48 last 749.79) - /unlink 50000 (time 1754010689.21 total 67.72 last 771.08) - /unlink 57382 (time 1754010699.21 total 77.72 last 738.05) - /unlink 60000 (time 1754010702.86 total 81.37 last 718.22) total: 64768 /unlink in 87.97 seconds: 736.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4624 (time 1754010738.63 total 10.00 last 462.38) - open/close 9335 (time 1754010748.64 total 20.00 last 471.02) - open/close 13965 (time 1754010758.64 total 30.00 last 462.91) - open/close 18618 (time 1754010768.64 total 40.01 last 465.26) - open/close 20000 (time 1754010771.68 total 43.05 last 454.31) - open/close 24624 (time 1754010781.68 total 53.05 last 462.30) - open/close 29231 (time 1754010791.68 total 63.05 last 460.69) - open/close 33864 (time 1754010801.68 total 73.05 last 463.27) - open/close 38456 (time 1754010811.69 total 83.05 last 459.11) - open/close 40000 (time 1754010814.95 total 86.32 last 472.72) - open/close 44674 (time 1754010824.95 total 96.32 last 467.28) - open/close 49045 (time 1754010834.96 total 106.32 last 437.02) - open/close 50000 (time 1754010837.18 total 108.54 last 429.90) - open/close 54394 (time 1754010847.18 total 118.54 last 439.37) - open/close 58939 (time 1754010857.18 total 128.54 last 454.50) - open/close 60000 (time 1754010859.63 total 130.99 last 433.05) - open/close 64522 (time 1754010869.63 total 141.00 last 452.14) total: 64768 open/close in 141.62 seconds: 457.34 ops/second - /unlink 7838 (time 1754010881.56 total 10.00 last 783.79) - /unlink 10000 (time 1754010884.26 total 12.70 last 801.29) - /unlink 17323 (time 1754010894.26 total 22.70 last 732.24) - /unlink 20000 (time 1754010897.52 total 25.96 last 821.38) - /unlink 26446 (time 1754010907.52 total 35.96 last 644.59) - /unlink 30000 (time 1754010912.27 total 40.71 last 748.19) - /unlink 37389 (time 1754010922.27 total 50.71 last 738.78) - /unlink 40000 (time 1754010925.50 total 53.94 last 807.80) - /unlink 47548 (time 1754010935.50 total 63.94 last 754.76) - /unlink 50000 (time 1754010938.81 total 67.26 last 740.07) - /unlink 58268 (time 1754010948.81 total 77.26 last 826.70) - /unlink 60000 (time 1754010951.15 total 79.60 last 740.21) total: 64768 /unlink in 85.58 seconds: 756.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2