== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:52:12 (1754009532) Stopping clients: oleg106-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg106-client.virtnet /mnt/lustre2 (opts:-f) oleg106-server: oleg106-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg106-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg106-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1691 (time 1754009604.50 total 10.01 last 169.01) - open/close 3755 (time 1754009614.51 total 20.01 last 206.24) - open/close 5747 (time 1754009624.51 total 30.02 last 199.14) - open/close 7682 (time 1754009634.51 total 40.02 last 193.50) - open/close 10000 (time 1754009643.17 total 48.68 last 267.49) - open/close 12583 (time 1754009653.17 total 58.68 last 258.29) - open/close 15261 (time 1754009663.18 total 68.68 last 267.74) - open/close 17987 (time 1754009673.18 total 78.69 last 272.58) - open/close 20000 (time 1754009680.03 total 85.54 last 293.74) - open/close 22780 (time 1754009690.03 total 95.54 last 277.95) - open/close 25442 (time 1754009700.03 total 105.54 last 266.15) - open/close 28147 (time 1754009710.03 total 115.54 last 270.50) - open/close 30000 (time 1754009715.36 total 120.86 last 348.23) - open/close 32496 (time 1754009725.36 total 130.87 last 249.44) - open/close 34682 (time 1754009735.36 total 140.87 last 218.57) - open/close 36602 (time 1754009745.37 total 150.87 last 191.96) - open/close 38976 (time 1754009755.37 total 160.88 last 237.32) - open/close 40000 (time 1754009758.87 total 164.38 last 292.11) - open/close 43172 (time 1754009768.88 total 174.39 last 317.07) - open/close 46005 (time 1754009778.88 total 184.39 last 283.25) - open/close 49429 (time 1754009788.88 total 194.39 last 342.40) - open/close 52788 (time 1754009798.88 total 204.39 last 335.87) - open/close 56476 (time 1754009808.88 total 214.39 last 368.74) - open/close 59301 (time 1754009818.88 total 224.39 last 282.45) - open/close 60000 (time 1754009821.10 total 226.61 last 315.44) - open/close 62998 (time 1754009831.10 total 236.61 last 299.74) total: 64768 open/close in 242.50 seconds: 267.09 ops/second - /unlink 6562 (time 1754009848.52 total 10.00 last 656.08) - /unlink 10000 (time 1754009855.24 total 16.72 last 511.60) - /unlink 15880 (time 1754009865.24 total 26.72 last 587.97) - /unlink 20000 (time 1754009871.98 total 33.46 last 611.51) - /unlink 25430 (time 1754009881.98 total 43.46 last 542.98) - /unlink 30000 (time 1754009888.88 total 50.36 last 662.22) - /unlink 36490 (time 1754009898.88 total 60.36 last 648.94) - /unlink 40000 (time 1754009904.04 total 65.52 last 680.21) - /unlink 48273 (time 1754009914.04 total 75.52 last 827.24) - /unlink 50000 (time 1754009916.30 total 77.78 last 764.51) - /unlink 58644 (time 1754009926.30 total 87.78 last 864.33) total: 64768 /unlink in 95.13 seconds: 680.81 ops/second Waiting for MDT destroys to complete oleg106-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 - open/close 4264 (time 1754009965.46 total 10.00 last 426.36) - open/close 9271 (time 1754009975.46 total 20.00 last 500.67) - open/close 13570 (time 1754009985.47 total 30.00 last 429.83) - open/close 17724 (time 1754009995.49 total 40.03 last 414.42) - open/close 20000 (time 1754010000.99 total 45.52 last 414.08) - open/close 24436 (time 1754010010.99 total 55.53 last 443.50) - open/close 28934 (time 1754010020.99 total 65.53 last 449.74) - open/close 30000 (time 1754010023.23 total 67.76 last 476.38) - open/close 35041 (time 1754010033.23 total 77.77 last 504.04) - open/close 40000 (time 1754010043.05 total 87.59 last 504.70) - open/close 45202 (time 1754010053.05 total 97.59 last 520.18) - open/close 50000 (time 1754010062.88 total 107.42 last 488.36) - open/close 54990 (time 1754010072.88 total 117.42 last 498.93) - open/close 59832 (time 1754010082.88 total 127.42 last 484.17) - open/close 64515 (time 1754010092.88 total 137.42 last 468.22) total: 64768 open/close in 137.94 seconds: 469.52 ops/second - /unlink 8788 (time 1754010104.73 total 10.00 last 878.74) - /unlink 17474 (time 1754010114.73 total 20.00 last 868.54) - /unlink 20000 (time 1754010117.81 total 23.07 last 822.09) - /unlink 29045 (time 1754010127.81 total 33.07 last 904.46) - /unlink 36995 (time 1754010137.81 total 43.07 last 794.97) - /unlink 40000 (time 1754010141.31 total 46.58 last 857.81) - /unlink 47954 (time 1754010151.31 total 56.58 last 795.35) - /unlink 50000 (time 1754010153.92 total 59.19 last 784.38) - /unlink 57507 (time 1754010163.92 total 69.19 last 750.63) - /unlink 60000 (time 1754010167.75 total 73.02 last 650.89) total: 64768 /unlink in 79.25 seconds: 817.27 ops/second Waiting for MDT destroys to complete oleg106-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 - open/close 5202 (time 1754010270.30 total 10.00 last 520.14) - open/close 10000 (time 1754010279.24 total 18.94 last 537.05) - open/close 15233 (time 1754010289.24 total 28.94 last 523.24) - open/close 19746 (time 1754010299.24 total 38.94 last 451.28) - open/close 24293 (time 1754010309.24 total 48.94 last 454.60) - open/close 28928 (time 1754010319.24 total 58.94 last 463.47) - open/close 30000 (time 1754010321.85 total 61.55 last 410.67) - open/close 34404 (time 1754010331.85 total 71.55 last 440.37) - open/close 38693 (time 1754010341.85 total 81.55 last 428.87) - open/close 40000 (time 1754010345.23 total 84.93 last 387.05) - open/close 44273 (time 1754010355.23 total 94.93 last 427.23) - open/close 48459 (time 1754010365.23 total 104.93 last 418.54) - open/close 50000 (time 1754010368.91 total 108.61 last 418.44) - open/close 54324 (time 1754010378.92 total 118.62 last 432.33) - open/close 58571 (time 1754010388.92 total 128.62 last 424.68) - open/close 60000 (time 1754010392.19 total 131.89 last 436.03) - open/close 64323 (time 1754010402.19 total 141.89 last 432.28) total: 64768 open/close in 142.93 seconds: 453.14 ops/second - /unlink 7447 (time 1754010414.68 total 10.00 last 744.64) - /unlink 10000 (time 1754010417.99 total 13.30 last 773.16) - /unlink 17751 (time 1754010427.99 total 23.30 last 775.08) - /unlink 20000 (time 1754010430.77 total 26.08 last 809.25) - /unlink 27828 (time 1754010440.77 total 36.08 last 782.71) - /unlink 30000 (time 1754010443.64 total 38.96 last 755.41) - /unlink 37744 (time 1754010453.64 total 48.96 last 774.33) - /unlink 40000 (time 1754010456.73 total 52.05 last 729.52) - /unlink 47992 (time 1754010466.74 total 62.05 last 799.16) - /unlink 50000 (time 1754010469.12 total 64.43 last 843.30) - /unlink 58347 (time 1754010479.12 total 74.43 last 834.63) total: 64768 /unlink in 81.88 seconds: 791.02 ops/second Waiting for MDT destroys to complete oleg106-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 - open/close 4631 (time 1754010537.05 total 10.00 last 463.03) - open/close 9208 (time 1754010547.05 total 20.00 last 457.63) - open/close 13694 (time 1754010557.05 total 30.00 last 448.51) - open/close 18297 (time 1754010567.06 total 40.01 last 460.20) - open/close 20000 (time 1754010570.73 total 43.69 last 462.97) - open/close 24659 (time 1754010580.74 total 53.69 last 465.88) - open/close 29060 (time 1754010590.74 total 63.69 last 440.02) - open/close 34351 (time 1754010600.74 total 73.69 last 529.09) - open/close 39782 (time 1754010610.74 total 83.69 last 543.00) - open/close 44936 (time 1754010620.74 total 93.69 last 515.36) - open/close 49443 (time 1754010630.74 total 103.69 last 450.63) - open/close 54049 (time 1754010640.76 total 113.72 last 459.54) - open/close 58797 (time 1754010650.77 total 123.72 last 474.71) - open/close 60000 (time 1754010653.32 total 126.27 last 471.55) - open/close 64684 (time 1754010663.32 total 136.27 last 468.33) total: 64768 open/close in 136.46 seconds: 474.64 ops/second - /unlink 7675 (time 1754010674.91 total 10.00 last 767.45) - /unlink 10000 (time 1754010677.87 total 12.96 last 785.57) - /unlink 17996 (time 1754010687.88 total 22.97 last 799.21) - /unlink 20000 (time 1754010690.57 total 25.66 last 742.84) - /unlink 27917 (time 1754010700.57 total 35.66 last 791.64) - /unlink 30000 (time 1754010703.04 total 38.13 last 843.27) - /unlink 37877 (time 1754010713.05 total 48.13 last 787.66) - /unlink 40000 (time 1754010715.81 total 50.90 last 767.13) - /unlink 48355 (time 1754010725.81 total 60.90 last 835.47) - /unlink 56577 (time 1754010735.81 total 70.90 last 822.16) - /unlink 60000 (time 1754010739.63 total 74.72 last 896.04) total: 64768 /unlink in 80.65 seconds: 803.06 ops/second Waiting for MDT destroys to complete oleg106-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 - open/close 4704 (time 1754010813.18 total 10.00 last 470.38) - open/close 9465 (time 1754010823.18 total 20.00 last 476.09) - open/close 14161 (time 1754010833.19 total 30.00 last 469.55) - open/close 18908 (time 1754010843.19 total 40.00 last 474.65) - open/close 20000 (time 1754010845.48 total 42.29 last 476.47) - open/close 24658 (time 1754010855.48 total 52.30 last 465.74) - open/close 29287 (time 1754010865.48 total 62.30 last 462.85) - open/close 33979 (time 1754010875.48 total 72.30 last 469.11) - open/close 38672 (time 1754010885.48 total 82.30 last 469.30) - open/close 40000 (time 1754010888.40 total 85.21 last 455.83) - open/close 44572 (time 1754010898.40 total 95.21 last 457.15) - open/close 49248 (time 1754010908.40 total 105.22 last 467.52) - open/close 53825 (time 1754010918.40 total 115.22 last 457.65) - open/close 58411 (time 1754010928.40 total 125.22 last 458.57) - open/close 60000 (time 1754010931.83 total 128.64 last 463.75) - open/close 64605 (time 1754010941.83 total 138.64 last 460.47) total: 64768 open/close in 138.98 seconds: 466.02 ops/second - /unlink 7348 (time 1754010953.48 total 10.00 last 734.73) - /unlink 10000 (time 1754010957.04 total 13.57 last 743.07) - /unlink 17406 (time 1754010967.04 total 23.57 last 740.59) - /unlink 20000 (time 1754010970.08 total 26.61 last 853.77) - /unlink 27680 (time 1754010980.08 total 36.61 last 767.95) - /unlink 30000 (time 1754010983.30 total 39.82 last 721.64) - /unlink 37664 (time 1754010993.30 total 49.82 last 766.32) - /unlink 40000 (time 1754010996.53 total 53.05 last 723.80) - /unlink 47583 (time 1754011006.53 total 63.05 last 758.30) - /unlink 50000 (time 1754011009.72 total 66.25 last 756.73) - /unlink 57216 (time 1754011019.72 total 76.25 last 721.59) - /unlink 60000 (time 1754011023.83 total 80.36 last 676.93) total: 64768 /unlink in 86.58 seconds: 748.07 ops/second Waiting for MDT destroys to complete oleg106-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg106-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2