== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:58:32 (1767034712) Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) oleg316-server: oleg316-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg316-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1308 (time 1767034768.02 total 10.00 last 130.79) - open/close 2549 (time 1767034778.02 total 20.00 last 124.08) - open/close 3899 (time 1767034788.03 total 30.01 last 134.85) - open/close 5049 (time 1767034798.03 total 40.02 last 114.93) - open/close 6073 (time 1767034808.04 total 50.02 last 102.36) - open/close 7026 (time 1767034818.04 total 60.03 last 95.26) - open/close 7914 (time 1767034828.05 total 70.04 last 88.73) - open/close 8848 (time 1767034838.05 total 80.04 last 93.38) - open/close 9840 (time 1767034848.06 total 90.04 last 99.16) - open/close 10808 (time 1767034858.07 total 100.06 last 96.66) - open/close 11754 (time 1767034868.08 total 110.06 last 94.55) - open/close 12818 (time 1767034878.08 total 120.07 last 106.36) - open/close 13948 (time 1767034888.09 total 130.07 last 112.93) - open/close 15274 (time 1767034898.09 total 140.07 last 132.58) - open/close 16294 (time 1767034908.09 total 150.08 last 101.95) - open/close 17436 (time 1767034918.10 total 160.09 last 114.10) - open/close 18759 (time 1767034928.10 total 170.09 last 132.26) - open/close 20000 (time 1767034936.81 total 178.80 last 142.46) - open/close 21112 (time 1767034946.82 total 188.81 last 111.15) - open/close 22321 (time 1767034956.82 total 198.81 last 120.88) - open/close 23875 (time 1767034966.83 total 208.81 last 155.31) - open/close 25541 (time 1767034976.83 total 218.82 last 166.54) - open/close 26901 (time 1767034986.84 total 228.82 last 135.94) - open/close 28511 (time 1767034996.84 total 238.82 last 160.93) - open/close 30000 (time 1767035005.86 total 247.85 last 165.07) - open/close 31781 (time 1767035015.86 total 257.85 last 178.07) - open/close 33748 (time 1767035025.86 total 267.85 last 196.70) - open/close 35857 (time 1767035035.86 total 277.85 last 210.86) - open/close 38120 (time 1767035045.87 total 287.85 last 226.25) - open/close 39780 (time 1767035055.87 total 297.86 last 165.93) - open/close 41549 (time 1767035065.87 total 307.86 last 176.88) - open/close 43163 (time 1767035075.87 total 317.86 last 161.40) - open/close 44847 (time 1767035085.88 total 327.86 last 168.32) - open/close 46738 (time 1767035095.88 total 337.86 last 189.05) - open/close 48794 (time 1767035105.88 total 347.86 last 205.59) - open/close 50000 (time 1767035110.55 total 352.53 last 258.33) - open/close 52109 (time 1767035120.55 total 362.54 last 210.85) - open/close 53823 (time 1767035130.55 total 372.54 last 171.33) - open/close 55427 (time 1767035140.56 total 382.54 last 160.36) - open/close 57019 (time 1767035150.56 total 392.55 last 159.09) - open/close 58668 (time 1767035160.56 total 402.55 last 164.88) - open/close 60000 (time 1767035168.33 total 410.31 last 171.56) - open/close 62102 (time 1767035178.33 total 420.32 last 210.17) - open/close 63887 (time 1767035188.33 total 430.32 last 178.45) total: 64768 open/close in 434.89 seconds: 148.93 ops/second - /unlink 4797 (time 1767035204.58 total 10.00 last 479.65) - /unlink 8834 (time 1767035214.59 total 20.00 last 403.63) - /unlink 10000 (time 1767035217.44 total 22.86 last 408.05) - /unlink 14445 (time 1767035227.44 total 32.86 last 444.48) - /unlink 19092 (time 1767035237.46 total 42.88 last 463.93) - /unlink 20000 (time 1767035239.69 total 45.11 last 407.16) - /unlink 24054 (time 1767035249.69 total 55.11 last 405.32) - /unlink 27126 (time 1767035259.70 total 65.11 last 307.07) - /unlink 30000 (time 1767035267.94 total 73.36 last 348.48) - /unlink 33711 (time 1767035277.95 total 83.36 last 371.02) - /unlink 37034 (time 1767035287.95 total 93.36 last 332.29) - /unlink 40000 (time 1767035297.48 total 102.90 last 311.09) - /unlink 43263 (time 1767035307.49 total 112.90 last 326.08) - /unlink 47019 (time 1767035317.49 total 122.91 last 375.54) - /unlink 50000 (time 1767035325.17 total 130.59 last 388.06) - /unlink 53711 (time 1767035335.17 total 140.59 last 371.09) - /unlink 57820 (time 1767035345.17 total 150.59 last 410.81) - /unlink 60000 (time 1767035351.30 total 156.72 last 355.79) - /unlink 63698 (time 1767035361.30 total 166.72 last 369.73) total: 64768 /unlink in 169.56 seconds: 381.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2215 (time 1767035394.79 total 10.00 last 221.44) - open/close 5299 (time 1767035404.79 total 20.00 last 308.39) - open/close 7851 (time 1767035414.79 total 30.00 last 255.20) - open/close 10000 (time 1767035422.61 total 37.82 last 274.79) - open/close 12964 (time 1767035432.61 total 47.83 last 296.36) - open/close 15842 (time 1767035442.61 total 57.83 last 287.78) - open/close 18825 (time 1767035452.61 total 67.83 last 298.29) - open/close 20000 (time 1767035456.59 total 71.81 last 294.80) - open/close 22875 (time 1767035466.60 total 81.82 last 287.36) - open/close 25624 (time 1767035476.60 total 91.82 last 274.83) - open/close 28527 (time 1767035486.60 total 101.82 last 290.25) - open/close 30000 (time 1767035490.89 total 106.11 last 343.28) - open/close 33556 (time 1767035500.90 total 116.11 last 355.52) - open/close 36041 (time 1767035510.90 total 126.12 last 248.43) - open/close 38526 (time 1767035520.90 total 136.12 last 248.49) - open/close 40000 (time 1767035526.20 total 141.42 last 278.09) - open/close 42677 (time 1767035536.20 total 151.42 last 267.68) - open/close 45567 (time 1767035546.20 total 161.42 last 288.96) - open/close 47957 (time 1767035556.21 total 171.42 last 238.91) - open/close 50000 (time 1767035565.90 total 181.11 last 210.84) - open/close 52663 (time 1767035575.90 total 191.12 last 266.23) - open/close 55575 (time 1767035585.90 total 201.12 last 291.19) - open/close 58096 (time 1767035595.90 total 211.12 last 252.05) - open/close 60000 (time 1767035602.66 total 217.88 last 281.67) - open/close 63108 (time 1767035612.66 total 227.88 last 310.70) total: 64768 open/close in 233.32 seconds: 277.59 ops/second - /unlink 5344 (time 1767035629.62 total 10.00 last 534.36) - /unlink 10000 (time 1767035637.08 total 17.46 last 624.07) - /unlink 16920 (time 1767035647.08 total 27.46 last 691.95) - /unlink 20000 (time 1767035650.58 total 30.96 last 880.46) - /unlink 27453 (time 1767035660.58 total 40.96 last 745.23) - /unlink 30000 (time 1767035663.94 total 44.32 last 758.41) - /unlink 37364 (time 1767035673.94 total 54.32 last 736.34) - /unlink 40000 (time 1767035677.35 total 57.73 last 772.81) - /unlink 47025 (time 1767035687.35 total 67.73 last 702.44) - /unlink 50000 (time 1767035691.73 total 72.11 last 679.62) - /unlink 56950 (time 1767035701.73 total 82.11 last 694.99) - /unlink 60000 (time 1767035706.63 total 87.01 last 622.78) total: 64768 /unlink in 95.05 seconds: 681.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4356 (time 1767035795.41 total 10.00 last 435.54) - open/close 8317 (time 1767035805.42 total 20.00 last 396.05) - open/close 10000 (time 1767035809.28 total 23.87 last 435.07) - open/close 14363 (time 1767035819.29 total 33.87 last 436.23) - open/close 18692 (time 1767035829.29 total 43.87 last 432.88) - open/close 20000 (time 1767035832.09 total 46.68 last 466.01) - open/close 23277 (time 1767035842.09 total 56.68 last 327.65) - open/close 27516 (time 1767035852.10 total 66.68 last 423.82) - open/close 30000 (time 1767035858.61 total 73.20 last 381.15) - open/close 34085 (time 1767035868.61 total 83.20 last 408.45) - open/close 37713 (time 1767035878.62 total 93.20 last 362.78) - open/close 40000 (time 1767035884.59 total 99.17 last 383.00) - open/close 43757 (time 1767035894.59 total 109.18 last 375.63) - open/close 47334 (time 1767035904.59 total 119.18 last 357.67) - open/close 50000 (time 1767035911.72 total 126.31 last 373.63) - open/close 53938 (time 1767035921.73 total 136.31 last 393.76) - open/close 57124 (time 1767035931.73 total 146.31 last 318.56) - open/close 60000 (time 1767035940.92 total 155.51 last 312.79) - open/close 63815 (time 1767035950.92 total 165.51 last 381.46) total: 64768 open/close in 168.00 seconds: 385.51 ops/second - /unlink 6522 (time 1767035964.96 total 10.00 last 652.14) - /unlink 10000 (time 1767035969.94 total 14.98 last 698.66) - /unlink 16935 (time 1767035979.94 total 24.98 last 693.45) - /unlink 20000 (time 1767035984.12 total 29.16 last 733.53) - /unlink 25586 (time 1767035994.12 total 39.16 last 558.56) - /unlink 30000 (time 1767036000.06 total 45.10 last 743.06) - /unlink 38082 (time 1767036010.06 total 55.10 last 808.13) - /unlink 40000 (time 1767036012.61 total 57.65 last 753.08) - /unlink 47922 (time 1767036022.61 total 67.65 last 792.20) - /unlink 50000 (time 1767036025.45 total 70.49 last 730.84) - /unlink 57337 (time 1767036035.45 total 80.49 last 733.69) - /unlink 60000 (time 1767036039.36 total 84.40 last 680.88) total: 64768 /unlink in 91.73 seconds: 706.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4361 (time 1767036076.36 total 10.00 last 436.01) - open/close 8512 (time 1767036086.36 total 20.00 last 415.06) - open/close 10000 (time 1767036089.57 total 23.21 last 463.96) - open/close 13321 (time 1767036099.57 total 33.21 last 332.02) - open/close 17762 (time 1767036109.57 total 43.21 last 444.04) - open/close 20000 (time 1767036114.27 total 47.91 last 476.28) - open/close 24811 (time 1767036124.27 total 57.91 last 481.05) - open/close 29560 (time 1767036134.27 total 67.92 last 474.81) - open/close 33850 (time 1767036144.28 total 77.92 last 428.87) - open/close 37201 (time 1767036154.28 total 87.92 last 335.07) - open/close 40000 (time 1767036163.59 total 97.24 last 300.46) - open/close 43530 (time 1767036173.59 total 107.24 last 352.93) - open/close 47451 (time 1767036183.59 total 117.24 last 392.08) - open/close 50000 (time 1767036190.67 total 124.31 last 360.50) - open/close 54105 (time 1767036200.67 total 134.31 last 410.46) - open/close 58748 (time 1767036210.67 total 144.31 last 464.22) - open/close 60000 (time 1767036213.30 total 146.94 last 476.52) - open/close 64512 (time 1767036223.30 total 156.94 last 451.14) total: 64768 open/close in 157.53 seconds: 411.14 ops/second - /unlink 8179 (time 1767036235.17 total 10.00 last 817.84) - /unlink 10000 (time 1767036237.41 total 12.24 last 811.46) - /unlink 18136 (time 1767036247.41 total 22.25 last 813.52) - /unlink 20000 (time 1767036249.96 total 24.79 last 732.47) - /unlink 26799 (time 1767036259.96 total 34.79 last 679.81) - /unlink 30000 (time 1767036263.93 total 38.77 last 805.33) - /unlink 36842 (time 1767036273.93 total 48.77 last 684.14) - /unlink 40000 (time 1767036278.18 total 53.01 last 743.88) - /unlink 47274 (time 1767036288.18 total 63.01 last 727.36) - /unlink 50000 (time 1767036292.08 total 66.92 last 698.39) - /unlink 57131 (time 1767036302.08 total 76.92 last 713.07) - /unlink 60000 (time 1767036305.68 total 80.52 last 797.27) total: 64768 /unlink in 87.37 seconds: 741.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4088 (time 1767036343.20 total 10.01 last 408.51) - open/close 8030 (time 1767036353.21 total 20.01 last 394.15) - open/close 10000 (time 1767036357.24 total 24.04 last 488.33) - open/close 14660 (time 1767036367.24 total 34.04 last 465.92) - open/close 19204 (time 1767036377.24 total 44.04 last 454.40) - open/close 23692 (time 1767036387.24 total 54.05 last 448.75) - open/close 27922 (time 1767036397.24 total 64.05 last 422.91) - open/close 30000 (time 1767036401.89 total 68.69 last 447.71) - open/close 33567 (time 1767036411.89 total 78.69 last 356.63) - open/close 37113 (time 1767036421.89 total 88.69 last 354.56) - open/close 40000 (time 1767036430.48 total 97.28 last 336.04) - open/close 42940 (time 1767036440.61 total 107.42 last 290.18) - open/close 46070 (time 1767036450.61 total 117.42 last 312.92) - open/close 48737 (time 1767036460.66 total 127.46 last 265.47) - open/close 50000 (time 1767036465.54 total 132.34 last 258.96) - open/close 52803 (time 1767036475.54 total 142.34 last 280.24) - open/close 56227 (time 1767036485.54 total 152.34 last 342.40) - open/close 59662 (time 1767036495.54 total 162.35 last 343.41) - open/close 63324 (time 1767036505.54 total 172.35 last 366.16) total: 64768 open/close in 176.34 seconds: 367.30 ops/second - /unlink 6587 (time 1767036521.01 total 10.00 last 658.64) - /unlink 10000 (time 1767036526.73 total 15.73 last 596.09) - /unlink 16106 (time 1767036536.73 total 25.73 last 610.54) - /unlink 20000 (time 1767036542.52 total 31.52 last 672.65) - /unlink 25888 (time 1767036552.52 total 41.52 last 588.74) - /unlink 30000 (time 1767036558.76 total 47.75 last 659.65) - /unlink 35144 (time 1767036568.76 total 57.75 last 514.40) - /unlink 40000 (time 1767036577.23 total 66.22 last 573.42) - /unlink 46375 (time 1767036587.23 total 76.22 last 637.46) - /unlink 50000 (time 1767036592.71 total 81.70 last 661.46) - /unlink 57223 (time 1767036602.71 total 91.70 last 722.24) - /unlink 60000 (time 1767036606.27 total 95.27 last 778.74) total: 64768 /unlink in 102.75 seconds: 630.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg316-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2