== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:57:28 (1755637048) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4631 (time 1755637094.12 total 10.00 last 463.05) - open/close 9304 (time 1755637104.12 total 20.00 last 467.29) - open/close 13992 (time 1755637114.12 total 30.00 last 468.73) - open/close 18515 (time 1755637124.12 total 40.00 last 452.28) - open/close 20000 (time 1755637127.33 total 43.21 last 463.04) - open/close 24636 (time 1755637137.33 total 53.21 last 463.58) - open/close 29263 (time 1755637147.33 total 63.21 last 462.64) - open/close 34031 (time 1755637157.33 total 73.21 last 476.78) - open/close 38672 (time 1755637167.33 total 83.21 last 464.01) - open/close 40000 (time 1755637170.32 total 86.20 last 444.45) - open/close 44539 (time 1755637180.32 total 96.20 last 453.83) - open/close 48865 (time 1755637190.32 total 106.21 last 432.54) - open/close 50000 (time 1755637193.02 total 108.90 last 421.20) - open/close 54466 (time 1755637203.02 total 118.90 last 446.53) - open/close 58932 (time 1755637213.02 total 128.90 last 446.57) - open/close 60000 (time 1755637215.41 total 131.29 last 446.71) - open/close 64436 (time 1755637225.41 total 141.29 last 443.53) total: 64768 open/close in 142.10 seconds: 455.78 ops/second - /unlink 7311 (time 1755637237.68 total 10.00 last 730.74) - /unlink 10000 (time 1755637240.94 total 13.27 last 824.25) - /unlink 17409 (time 1755637250.94 total 23.27 last 740.88) - /unlink 20000 (time 1755637254.67 total 27.00 last 694.10) - /unlink 27554 (time 1755637264.68 total 37.00 last 755.33) - /unlink 30000 (time 1755637267.63 total 39.96 last 827.38) - /unlink 37283 (time 1755637277.63 total 49.96 last 728.23) - /unlink 40000 (time 1755637280.91 total 53.24 last 829.24) - /unlink 47196 (time 1755637290.91 total 63.24 last 719.56) - /unlink 50000 (time 1755637294.92 total 67.25 last 699.13) - /unlink 57757 (time 1755637304.92 total 77.25 last 775.66) - /unlink 60000 (time 1755637307.14 total 79.47 last 1009.70) total: 64768 /unlink in 85.95 seconds: 753.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4704 (time 1755637344.40 total 10.00 last 470.31) - open/close 9356 (time 1755637354.40 total 20.01 last 465.01) - open/close 14014 (time 1755637364.41 total 30.01 last 465.74) - open/close 18559 (time 1755637374.41 total 40.01 last 454.48) - open/close 20000 (time 1755637377.47 total 43.07 last 470.43) - open/close 24730 (time 1755637387.47 total 53.07 last 472.93) - open/close 29371 (time 1755637397.47 total 63.07 last 464.10) - open/close 33985 (time 1755637407.47 total 73.07 last 461.33) - open/close 38629 (time 1755637417.47 total 83.07 last 464.38) - open/close 40000 (time 1755637420.31 total 85.92 last 482.50) - open/close 44481 (time 1755637430.32 total 95.92 last 448.08) - open/close 49082 (time 1755637440.32 total 105.92 last 460.09) - open/close 50000 (time 1755637442.38 total 107.98 last 445.52) - open/close 54480 (time 1755637452.38 total 117.98 last 447.96) - open/close 58791 (time 1755637462.38 total 127.98 last 431.02) - open/close 60000 (time 1755637465.09 total 130.69 last 446.29) - open/close 64378 (time 1755637475.09 total 140.69 last 437.72) total: 64768 open/close in 141.68 seconds: 457.16 ops/second - /unlink 7430 (time 1755637487.48 total 10.00 last 742.96) - /unlink 10000 (time 1755637490.60 total 13.12 last 823.01) - /unlink 17493 (time 1755637500.60 total 23.12 last 749.27) - /unlink 20000 (time 1755637504.19 total 26.71 last 698.52) - /unlink 27390 (time 1755637514.20 total 36.72 last 738.33) - /unlink 30000 (time 1755637517.49 total 40.02 last 792.18) - /unlink 37947 (time 1755637527.49 total 50.02 last 794.66) - /unlink 40000 (time 1755637530.37 total 52.90 last 712.76) - /unlink 48211 (time 1755637540.37 total 62.90 last 820.96) - /unlink 50000 (time 1755637542.60 total 65.13 last 803.56) - /unlink 58166 (time 1755637552.60 total 75.13 last 816.52) - /unlink 60000 (time 1755637554.67 total 77.19 last 888.96) total: 64768 /unlink in 83.14 seconds: 779.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4520 (time 1755637643.24 total 10.00 last 451.97) - open/close 9266 (time 1755637653.24 total 20.00 last 474.59) - open/close 13627 (time 1755637663.24 total 30.00 last 436.07) - open/close 18407 (time 1755637673.25 total 40.00 last 477.91) - open/close 20000 (time 1755637676.54 total 43.29 last 484.03) - open/close 24725 (time 1755637686.54 total 53.30 last 472.47) - open/close 29448 (time 1755637696.54 total 63.30 last 472.24) - open/close 34069 (time 1755637706.54 total 73.30 last 462.05) - open/close 38599 (time 1755637716.54 total 83.30 last 453.00) - open/close 40000 (time 1755637719.61 total 86.37 last 456.42) - open/close 44683 (time 1755637729.61 total 96.37 last 468.24) - open/close 49335 (time 1755637739.61 total 106.37 last 465.15) - open/close 54071 (time 1755637749.61 total 116.37 last 473.57) - open/close 58541 (time 1755637759.61 total 126.37 last 446.98) - open/close 60000 (time 1755637763.02 total 129.78 last 427.71) - open/close 64409 (time 1755637773.03 total 139.78 last 440.87) total: 64768 open/close in 140.64 seconds: 460.53 ops/second - /unlink 7521 (time 1755637785.20 total 10.00 last 752.06) - /unlink 10000 (time 1755637788.18 total 12.98 last 831.26) - /unlink 17417 (time 1755637798.18 total 22.98 last 741.70) - /unlink 20000 (time 1755637801.74 total 26.55 last 724.74) - /unlink 27671 (time 1755637811.74 total 36.55 last 767.03) - /unlink 30000 (time 1755637814.99 total 39.79 last 717.27) - /unlink 37581 (time 1755637824.99 total 49.79 last 758.08) - /unlink 40000 (time 1755637828.13 total 52.94 last 769.68) - /unlink 47464 (time 1755637838.14 total 62.94 last 746.36) - /unlink 50000 (time 1755637841.67 total 66.47 last 717.26) - /unlink 57546 (time 1755637851.67 total 76.48 last 754.52) - /unlink 60000 (time 1755637854.56 total 79.36 last 849.39) total: 64768 /unlink in 85.75 seconds: 755.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4384 (time 1755637892.17 total 10.00 last 438.36) - open/close 8543 (time 1755637902.17 total 20.00 last 415.89) - open/close 10000 (time 1755637905.21 total 23.04 last 479.15) - open/close 14601 (time 1755637915.21 total 33.04 last 460.07) - open/close 19308 (time 1755637925.21 total 43.04 last 470.64) - open/close 23465 (time 1755637935.21 total 53.04 last 415.69) - open/close 27516 (time 1755637945.21 total 63.05 last 405.02) - open/close 30000 (time 1755637950.51 total 68.35 last 468.71) - open/close 34752 (time 1755637960.51 total 78.35 last 475.11) - open/close 39178 (time 1755637970.51 total 88.35 last 442.52) - open/close 43669 (time 1755637980.52 total 98.35 last 449.06) - open/close 47813 (time 1755637990.52 total 108.35 last 414.37) - open/close 50000 (time 1755637995.84 total 113.67 last 411.19) - open/close 54445 (time 1755638005.84 total 123.67 last 444.49) - open/close 58463 (time 1755638015.84 total 133.67 last 401.69) - open/close 60000 (time 1755638019.37 total 137.20 last 435.39) - open/close 63984 (time 1755638029.37 total 147.20 last 398.32) total: 64768 open/close in 149.21 seconds: 434.06 ops/second - /unlink 6384 (time 1755638042.78 total 10.00 last 638.40) - /unlink 10000 (time 1755638047.74 total 14.96 last 729.77) - /unlink 16406 (time 1755638057.74 total 24.96 last 640.47) - /unlink 20000 (time 1755638062.56 total 29.78 last 745.80) - /unlink 27498 (time 1755638072.57 total 39.78 last 749.18) - /unlink 30000 (time 1755638076.02 total 43.24 last 723.64) - /unlink 37326 (time 1755638086.02 total 53.24 last 732.56) - /unlink 40000 (time 1755638089.77 total 56.99 last 714.32) - /unlink 47341 (time 1755638099.77 total 66.99 last 734.10) - /unlink 50000 (time 1755638102.77 total 69.99 last 884.23) - /unlink 57806 (time 1755638112.78 total 79.99 last 780.54) - /unlink 60000 (time 1755638115.88 total 83.10 last 705.99) total: 64768 /unlink in 89.27 seconds: 725.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4656 (time 1755638156.33 total 10.00 last 465.55) - open/close 9390 (time 1755638166.33 total 20.00 last 473.33) - open/close 14120 (time 1755638176.33 total 30.00 last 472.95) - open/close 18756 (time 1755638186.33 total 40.00 last 463.55) - open/close 20000 (time 1755638189.10 total 42.78 last 449.02) - open/close 24339 (time 1755638199.10 total 52.78 last 433.79) - open/close 28689 (time 1755638209.11 total 62.78 last 434.93) - open/close 30000 (time 1755638211.98 total 65.65 last 456.70) - open/close 34778 (time 1755638221.98 total 75.65 last 477.79) - open/close 39280 (time 1755638231.98 total 85.65 last 450.18) - open/close 43670 (time 1755638241.98 total 95.65 last 438.94) - open/close 47731 (time 1755638251.98 total 105.65 last 406.08) - open/close 50000 (time 1755638257.88 total 111.55 last 384.69) - open/close 54267 (time 1755638267.88 total 121.55 last 426.68) - open/close 58663 (time 1755638277.88 total 131.55 last 439.60) - open/close 60000 (time 1755638280.98 total 134.65 last 430.80) - open/close 64522 (time 1755638290.98 total 144.65 last 452.19) total: 64768 open/close in 145.17 seconds: 446.16 ops/second - /unlink 7841 (time 1755638302.87 total 10.00 last 784.03) - /unlink 10000 (time 1755638305.53 total 12.66 last 813.01) - /unlink 17725 (time 1755638315.53 total 22.66 last 772.41) - /unlink 20000 (time 1755638318.80 total 25.93 last 695.56) - /unlink 27754 (time 1755638328.80 total 35.93 last 775.33) - /unlink 30000 (time 1755638331.33 total 38.45 last 889.86) - /unlink 37588 (time 1755638341.33 total 48.45 last 758.76) - /unlink 40000 (time 1755638345.65 total 52.78 last 557.49) - /unlink 47264 (time 1755638355.65 total 62.78 last 726.38) - /unlink 50000 (time 1755638359.43 total 66.56 last 723.66) - /unlink 57888 (time 1755638369.43 total 76.56 last 788.80) - /unlink 60000 (time 1755638371.78 total 78.91 last 898.94) total: 64768 /unlink in 84.87 seconds: 763.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2