== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:37:30 (1763876250) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5133 (time 1763876296.70 total 10.00 last 513.23) - open/close 10000 (time 1763876306.45 total 19.76 last 498.99) - open/close 15054 (time 1763876316.45 total 29.76 last 505.33) - open/close 19762 (time 1763876326.45 total 39.76 last 470.66) - open/close 24609 (time 1763876336.45 total 49.76 last 484.69) - open/close 29157 (time 1763876346.45 total 59.76 last 454.76) - open/close 33884 (time 1763876356.46 total 69.76 last 472.68) - open/close 38763 (time 1763876366.46 total 79.76 last 487.90) - open/close 40000 (time 1763876369.12 total 82.43 last 463.38) - open/close 44690 (time 1763876379.13 total 92.43 last 468.96) - open/close 49268 (time 1763876389.13 total 102.43 last 457.72) - open/close 54243 (time 1763876399.13 total 112.43 last 497.42) - open/close 58979 (time 1763876409.13 total 122.44 last 473.50) - open/close 60000 (time 1763876411.23 total 124.53 last 486.75) total: 64768 open/close in 134.21 seconds: 482.59 ops/second - /unlink 8611 (time 1763876432.13 total 10.00 last 861.08) - /unlink 15953 (time 1763876442.13 total 20.00 last 734.15) - /unlink 20000 (time 1763876448.60 total 26.47 last 625.56) - /unlink 27548 (time 1763876458.60 total 36.47 last 754.79) - /unlink 30000 (time 1763876461.34 total 39.21 last 895.07) - /unlink 37670 (time 1763876471.34 total 49.21 last 766.91) - /unlink 40000 (time 1763876474.11 total 51.98 last 841.39) - /unlink 47884 (time 1763876484.11 total 61.98 last 788.34) - /unlink 50000 (time 1763876487.00 total 64.87 last 732.65) - /unlink 57651 (time 1763876497.00 total 74.87 last 765.09) - /unlink 60000 (time 1763876499.83 total 77.70 last 829.21) total: 64768 /unlink in 84.40 seconds: 767.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4789 (time 1763876542.04 total 10.00 last 478.87) - open/close 9308 (time 1763876552.04 total 20.00 last 451.87) - open/close 14155 (time 1763876562.04 total 30.00 last 484.68) - open/close 18608 (time 1763876572.04 total 40.00 last 445.22) - open/close 20000 (time 1763876574.67 total 42.63 last 530.03) - open/close 25237 (time 1763876584.67 total 52.63 last 523.62) - open/close 30000 (time 1763876594.54 total 62.50 last 482.56) - open/close 34633 (time 1763876604.55 total 72.51 last 462.78) - open/close 38454 (time 1763876614.55 total 82.51 last 382.04) - open/close 40000 (time 1763876618.46 total 86.42 last 396.00) - open/close 43903 (time 1763876628.46 total 96.42 last 390.30) - open/close 48256 (time 1763876638.46 total 106.42 last 435.29) - open/close 50000 (time 1763876642.80 total 110.76 last 401.53) - open/close 53653 (time 1763876652.80 total 120.76 last 365.29) - open/close 58228 (time 1763876662.80 total 130.76 last 457.39) - open/close 60000 (time 1763876666.88 total 134.84 last 434.66) - open/close 64574 (time 1763876676.88 total 144.84 last 457.39) total: 64768 open/close in 145.41 seconds: 445.40 ops/second - /unlink 7275 (time 1763876688.85 total 10.00 last 727.46) - /unlink 10000 (time 1763876692.31 total 13.46 last 788.42) - /unlink 17807 (time 1763876702.31 total 23.46 last 780.63) - /unlink 20000 (time 1763876705.72 total 26.87 last 643.06) - /unlink 27494 (time 1763876715.72 total 36.87 last 749.38) - /unlink 30000 (time 1763876719.34 total 40.49 last 691.51) - /unlink 37370 (time 1763876729.35 total 50.49 last 736.95) - /unlink 40000 (time 1763876732.66 total 53.81 last 792.71) - /unlink 47800 (time 1763876742.66 total 63.81 last 779.87) - /unlink 50000 (time 1763876745.50 total 66.65 last 775.36) - /unlink 58009 (time 1763876755.50 total 76.65 last 800.90) - /unlink 60000 (time 1763876757.71 total 78.86 last 900.07) total: 64768 /unlink in 85.07 seconds: 761.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4768 (time 1763876796.35 total 10.00 last 476.78) - open/close 9362 (time 1763876806.35 total 20.00 last 459.38) - open/close 14419 (time 1763876816.35 total 30.00 last 505.68) - open/close 19618 (time 1763876826.35 total 40.00 last 519.83) - open/close 25029 (time 1763876836.35 total 50.00 last 541.04) - open/close 30000 (time 1763876846.27 total 59.92 last 501.46) - open/close 35008 (time 1763876856.27 total 69.92 last 500.71) - open/close 40000 (time 1763876866.26 total 79.91 last 499.82) - open/close 44219 (time 1763876876.26 total 89.91 last 421.84) - open/close 48919 (time 1763876886.26 total 99.91 last 469.98) - open/close 50000 (time 1763876888.56 total 102.21 last 470.09) - open/close 55056 (time 1763876898.56 total 112.21 last 505.60) - open/close 59800 (time 1763876908.56 total 122.21 last 474.36) - open/close 64397 (time 1763876918.56 total 132.21 last 459.63) total: 64768 open/close in 133.16 seconds: 486.38 ops/second - /unlink 6680 (time 1763876930.85 total 10.00 last 667.96) - /unlink 10000 (time 1763876935.63 total 14.78 last 694.74) - /unlink 17047 (time 1763876945.63 total 24.78 last 704.65) - /unlink 20000 (time 1763876949.42 total 28.56 last 780.21) - /unlink 27209 (time 1763876959.42 total 38.57 last 720.82) - /unlink 30000 (time 1763876963.19 total 42.34 last 740.01) - /unlink 37365 (time 1763876973.19 total 52.34 last 736.46) - /unlink 40000 (time 1763876976.35 total 55.49 last 835.12) - /unlink 47609 (time 1763876986.35 total 65.49 last 760.84) - /unlink 50000 (time 1763876989.10 total 68.25 last 867.90) - /unlink 58133 (time 1763876999.10 total 78.25 last 813.27) total: 64768 /unlink in 86.43 seconds: 749.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4869 (time 1763877045.18 total 10.00 last 486.85) - open/close 9531 (time 1763877055.18 total 20.00 last 466.10) - open/close 14614 (time 1763877065.18 total 30.00 last 508.26) - open/close 19621 (time 1763877075.18 total 40.01 last 500.60) - open/close 24910 (time 1763877085.18 total 50.01 last 528.88) - open/close 30000 (time 1763877094.73 total 59.55 last 533.12) - open/close 35294 (time 1763877104.73 total 69.55 last 529.38) - open/close 40000 (time 1763877113.27 total 78.09 last 551.16) - open/close 44560 (time 1763877123.27 total 88.09 last 455.96) - open/close 48981 (time 1763877133.27 total 98.10 last 442.02) - open/close 50000 (time 1763877135.59 total 100.41 last 439.43) - open/close 54123 (time 1763877145.59 total 110.41 last 412.27) - open/close 59195 (time 1763877155.59 total 120.42 last 507.13) - open/close 63563 (time 1763877165.59 total 130.42 last 436.72) total: 64768 open/close in 133.22 seconds: 486.17 ops/second - /unlink 7287 (time 1763877179.75 total 10.00 last 728.56) - /unlink 10000 (time 1763877183.18 total 13.43 last 792.20) - /unlink 17308 (time 1763877193.18 total 23.43 last 730.78) - /unlink 20000 (time 1763877196.87 total 27.12 last 728.62) - /unlink 26724 (time 1763877206.87 total 37.12 last 672.34) - /unlink 30000 (time 1763877211.37 total 41.62 last 728.70) - /unlink 37310 (time 1763877221.37 total 51.62 last 730.94) - /unlink 40000 (time 1763877224.65 total 54.90 last 820.33) - /unlink 47091 (time 1763877234.65 total 64.90 last 709.09) - /unlink 50000 (time 1763877238.65 total 68.90 last 727.65) - /unlink 57198 (time 1763877248.65 total 78.90 last 719.76) - /unlink 60000 (time 1763877252.56 total 82.81 last 716.44) total: 64768 /unlink in 89.10 seconds: 726.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4560 (time 1763877289.29 total 10.00 last 455.99) - open/close 9137 (time 1763877299.29 total 20.00 last 457.63) - open/close 13597 (time 1763877309.29 total 30.00 last 445.94) - open/close 18153 (time 1763877319.30 total 40.01 last 455.52) - open/close 20000 (time 1763877323.31 total 44.02 last 459.95) - open/close 24597 (time 1763877333.31 total 54.02 last 459.63) - open/close 29206 (time 1763877343.31 total 64.02 last 460.90) - open/close 33853 (time 1763877353.32 total 74.02 last 464.63) - open/close 38359 (time 1763877363.32 total 84.02 last 450.59) - open/close 40000 (time 1763877367.05 total 87.76 last 439.24) - open/close 44328 (time 1763877377.05 total 97.76 last 432.75) - open/close 48605 (time 1763877387.05 total 107.76 last 427.66) - open/close 50000 (time 1763877390.24 total 110.95 last 437.68) - open/close 54345 (time 1763877400.24 total 120.95 last 434.48) - open/close 58708 (time 1763877410.24 total 130.95 last 436.22) - open/close 60000 (time 1763877413.26 total 133.97 last 428.04) - open/close 64274 (time 1763877423.26 total 143.97 last 427.31) total: 64768 open/close in 145.32 seconds: 445.68 ops/second - /unlink 7003 (time 1763877436.05 total 10.00 last 700.30) - /unlink 10000 (time 1763877440.27 total 14.22 last 709.75) - /unlink 17321 (time 1763877450.30 total 24.25 last 730.33) - /unlink 20000 (time 1763877454.35 total 28.30 last 661.40) - /unlink 27051 (time 1763877464.35 total 38.30 last 705.09) - /unlink 30000 (time 1763877468.58 total 42.53 last 697.09) - /unlink 37258 (time 1763877478.58 total 52.53 last 725.75) - /unlink 40000 (time 1763877482.52 total 56.47 last 694.90) - /unlink 47501 (time 1763877492.52 total 66.48 last 750.01) - /unlink 50000 (time 1763877495.56 total 69.51 last 822.80) - /unlink 57124 (time 1763877505.57 total 79.52 last 711.68) - /unlink 60000 (time 1763877509.63 total 83.58 last 709.44) total: 64768 /unlink in 90.26 seconds: 717.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2