== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:33:18 (1756675998) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4057 (time 1756676040.80 total 10.00 last 405.64) - open/close 9036 (time 1756676050.80 total 20.00 last 497.81) - open/close 13792 (time 1756676060.81 total 30.00 last 475.52) - open/close 18138 (time 1756676070.81 total 40.01 last 434.57) - open/close 20000 (time 1756676074.66 total 43.85 last 483.72) - open/close 24727 (time 1756676084.66 total 53.86 last 472.67) - open/close 29802 (time 1756676094.66 total 63.86 last 507.42) - open/close 34403 (time 1756676104.66 total 73.86 last 460.05) - open/close 38932 (time 1756676114.66 total 83.86 last 452.83) - open/close 40000 (time 1756676117.67 total 86.87 last 354.78) - open/close 43941 (time 1756676127.67 total 96.87 last 394.05) - open/close 48402 (time 1756676137.67 total 106.87 last 446.03) - open/close 50000 (time 1756676141.55 total 110.75 last 412.19) - open/close 54691 (time 1756676151.57 total 120.77 last 467.95) - open/close 59479 (time 1756676161.65 total 130.85 last 475.27) - open/close 63934 (time 1756676171.65 total 140.85 last 445.44) total: 64768 open/close in 142.79 seconds: 453.59 ops/second - /unlink 7282 (time 1756676185.07 total 10.00 last 728.17) - /unlink 10000 (time 1756676188.39 total 13.32 last 819.81) - /unlink 16854 (time 1756676198.39 total 23.32 last 685.34) - /unlink 20000 (time 1756676202.93 total 27.86 last 692.70) - /unlink 28273 (time 1756676212.93 total 37.86 last 827.05) - /unlink 36888 (time 1756676222.93 total 47.86 last 861.46) - /unlink 40000 (time 1756676226.91 total 51.83 last 783.54) - /unlink 47758 (time 1756676236.91 total 61.83 last 775.74) - /unlink 50000 (time 1756676240.34 total 65.27 last 652.24) - /unlink 57007 (time 1756676250.34 total 75.27 last 700.68) - /unlink 60000 (time 1756676254.61 total 79.54 last 701.98) total: 64768 /unlink in 85.49 seconds: 757.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4553 (time 1756676290.90 total 10.00 last 455.28) - open/close 8962 (time 1756676300.90 total 20.00 last 440.87) - open/close 10000 (time 1756676303.21 total 22.31 last 449.82) - open/close 14752 (time 1756676313.21 total 32.31 last 475.11) - open/close 20000 (time 1756676323.06 total 42.17 last 532.55) - open/close 25142 (time 1756676333.06 total 52.17 last 514.12) - open/close 29973 (time 1756676343.06 total 62.17 last 483.03) - open/close 34737 (time 1756676353.07 total 72.17 last 476.34) - open/close 39772 (time 1756676363.07 total 82.17 last 503.39) - open/close 44524 (time 1756676373.07 total 92.17 last 475.13) - open/close 49199 (time 1756676383.07 total 102.17 last 467.44) - open/close 54190 (time 1756676393.07 total 112.17 last 499.06) - open/close 58776 (time 1756676403.07 total 122.18 last 458.58) - open/close 60000 (time 1756676405.64 total 124.75 last 476.10) total: 64768 open/close in 134.71 seconds: 480.80 ops/second - /unlink 7326 (time 1756676426.93 total 10.00 last 732.55) - /unlink 10000 (time 1756676430.56 total 13.63 last 737.05) - /unlink 17370 (time 1756676440.56 total 23.63 last 736.90) - /unlink 20000 (time 1756676444.13 total 27.20 last 737.22) - /unlink 27307 (time 1756676454.13 total 37.20 last 730.64) - /unlink 30000 (time 1756676457.90 total 40.96 last 715.34) - /unlink 36912 (time 1756676467.90 total 50.96 last 691.15) - /unlink 40000 (time 1756676472.10 total 55.16 last 735.31) - /unlink 47562 (time 1756676482.10 total 65.16 last 756.18) - /unlink 50000 (time 1756676485.73 total 68.79 last 671.51) - /unlink 57053 (time 1756676495.73 total 78.80 last 705.19) - /unlink 60000 (time 1756676500.08 total 83.14 last 677.87) total: 64768 /unlink in 89.52 seconds: 723.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4643 (time 1756676535.45 total 10.00 last 464.25) - open/close 9221 (time 1756676545.45 total 20.00 last 457.75) - open/close 13498 (time 1756676555.45 total 30.00 last 427.63) - open/close 16548 (time 1756676565.45 total 40.00 last 304.99) - open/close 20000 (time 1756676573.81 total 48.36 last 413.19) - open/close 24146 (time 1756676583.81 total 58.36 last 414.46) - open/close 28398 (time 1756676593.81 total 68.36 last 425.19) - open/close 30000 (time 1756676597.55 total 72.10 last 428.52) - open/close 34623 (time 1756676607.55 total 82.10 last 462.23) - open/close 39115 (time 1756676617.55 total 92.10 last 449.18) - open/close 43620 (time 1756676627.55 total 102.10 last 450.49) - open/close 48084 (time 1756676637.55 total 112.10 last 446.34) - open/close 50000 (time 1756676641.76 total 116.32 last 454.68) - open/close 54448 (time 1756676651.77 total 126.32 last 444.80) - open/close 58667 (time 1756676661.77 total 136.32 last 421.84) - open/close 60000 (time 1756676664.79 total 139.35 last 440.57) - open/close 64447 (time 1756676674.79 total 149.35 last 444.67) total: 64768 open/close in 150.00 seconds: 431.79 ops/second - /unlink 7514 (time 1756676686.89 total 10.00 last 751.38) - /unlink 10000 (time 1756676690.36 total 13.47 last 716.63) - /unlink 17408 (time 1756676700.36 total 23.47 last 740.74) - /unlink 20000 (time 1756676703.54 total 26.65 last 814.34) - /unlink 27143 (time 1756676713.55 total 36.65 last 714.24) - /unlink 30000 (time 1756676717.53 total 40.64 last 717.62) - /unlink 37426 (time 1756676727.53 total 50.64 last 742.59) - /unlink 40000 (time 1756676731.12 total 54.23 last 717.01) - /unlink 47214 (time 1756676741.12 total 64.23 last 721.34) - /unlink 50000 (time 1756676744.69 total 67.80 last 779.57) - /unlink 58082 (time 1756676754.69 total 77.80 last 808.17) - /unlink 60000 (time 1756676757.62 total 80.73 last 655.10) total: 64768 /unlink in 87.63 seconds: 739.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4655 (time 1756676799.15 total 10.00 last 465.44) - open/close 9255 (time 1756676809.15 total 20.00 last 459.99) - open/close 10000 (time 1756676811.33 total 22.18 last 342.29) - open/close 14182 (time 1756676821.33 total 32.18 last 418.18) - open/close 18882 (time 1756676831.33 total 42.18 last 469.96) - open/close 20000 (time 1756676833.78 total 44.63 last 456.33) - open/close 24376 (time 1756676843.78 total 54.63 last 437.52) - open/close 28852 (time 1756676853.78 total 64.63 last 447.53) - open/close 30000 (time 1756676856.23 total 67.08 last 468.45) - open/close 33992 (time 1756676866.23 total 77.09 last 399.12) - open/close 38551 (time 1756676876.23 total 87.09 last 455.88) - open/close 40000 (time 1756676879.37 total 90.23 last 461.42) - open/close 44490 (time 1756676889.37 total 100.23 last 448.97) - open/close 48932 (time 1756676899.38 total 110.23 last 444.17) - open/close 50000 (time 1756676901.87 total 112.73 last 427.46) - open/close 54012 (time 1756676911.88 total 122.73 last 401.14) - open/close 58421 (time 1756676921.88 total 132.73 last 440.86) - open/close 60000 (time 1756676925.75 total 136.60 last 407.77) - open/close 64448 (time 1756676935.83 total 146.69 last 441.06) total: 64768 open/close in 147.39 seconds: 439.44 ops/second - /unlink 7751 (time 1756676947.89 total 10.00 last 775.05) - /unlink 10000 (time 1756676951.24 total 13.35 last 671.15) - /unlink 16798 (time 1756676961.24 total 23.35 last 679.75) - /unlink 20000 (time 1756676965.74 total 27.85 last 712.17) - /unlink 27395 (time 1756676975.74 total 37.85 last 739.45) - /unlink 30000 (time 1756676979.11 total 41.21 last 774.12) - /unlink 37583 (time 1756676989.11 total 51.22 last 757.86) - /unlink 40000 (time 1756676992.41 total 54.51 last 733.83) - /unlink 47314 (time 1756677002.41 total 64.51 last 731.35) - /unlink 50000 (time 1756677006.26 total 68.37 last 697.27) - /unlink 56898 (time 1756677016.26 total 78.37 last 689.78) - /unlink 60000 (time 1756677020.63 total 82.74 last 709.58) total: 64768 /unlink in 89.30 seconds: 725.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4440 (time 1756677057.54 total 10.00 last 443.97) - open/close 8815 (time 1756677067.54 total 20.00 last 437.47) - open/close 10000 (time 1756677070.25 total 22.71 last 437.36) - open/close 14174 (time 1756677080.25 total 32.71 last 417.32) - open/close 18924 (time 1756677090.25 total 42.71 last 474.92) - open/close 20000 (time 1756677092.50 total 44.97 last 477.55) - open/close 24667 (time 1756677102.50 total 54.97 last 466.66) - open/close 29322 (time 1756677112.51 total 64.97 last 465.44) - open/close 33892 (time 1756677122.51 total 74.97 last 456.91) - open/close 38509 (time 1756677132.51 total 84.97 last 461.70) - open/close 40000 (time 1756677135.77 total 88.24 last 456.45) - open/close 44589 (time 1756677145.77 total 98.24 last 458.88) - open/close 49036 (time 1756677155.78 total 108.24 last 444.66) - open/close 50000 (time 1756677157.99 total 110.45 last 435.97) - open/close 54456 (time 1756677167.99 total 120.45 last 445.53) - open/close 58892 (time 1756677177.99 total 130.45 last 443.53) - open/close 60000 (time 1756677180.43 total 132.89 last 454.76) - open/close 64416 (time 1756677190.43 total 142.89 last 441.59) total: 64768 open/close in 143.65 seconds: 450.88 ops/second - /unlink 7309 (time 1756677202.64 total 10.00 last 730.88) - /unlink 10000 (time 1756677206.41 total 13.77 last 714.19) - /unlink 17381 (time 1756677216.41 total 23.77 last 738.03) - /unlink 20000 (time 1756677220.04 total 27.40 last 720.93) - /unlink 27482 (time 1756677230.04 total 37.40 last 748.16) - /unlink 30000 (time 1756677233.22 total 40.58 last 791.57) - /unlink 37213 (time 1756677243.22 total 50.58 last 721.22) - /unlink 40000 (time 1756677246.74 total 54.10 last 792.13) - /unlink 47031 (time 1756677256.74 total 64.10 last 703.02) - /unlink 50000 (time 1756677260.42 total 67.79 last 806.28) - /unlink 57605 (time 1756677270.42 total 77.79 last 760.48) - /unlink 60000 (time 1756677273.72 total 81.08 last 727.49) total: 64768 /unlink in 87.23 seconds: 742.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2