== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:10:20 (1758737420) Stopping clients: oleg218-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg218-client.virtnet /mnt/lustre2 (opts:-f) oleg218-server: oleg218-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg218-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 838 (time 1758737538.56 total 10.01 last 83.69) - open/close 1753 (time 1758737548.57 total 20.02 last 91.47) - open/close 2744 (time 1758737558.58 total 30.03 last 99.01) - open/close 3677 (time 1758737568.58 total 40.03 last 93.27) - open/close 4592 (time 1758737578.58 total 50.03 last 91.49) - open/close 5472 (time 1758737588.59 total 60.04 last 87.93) - open/close 6352 (time 1758737598.60 total 70.05 last 87.92) - open/close 7261 (time 1758737608.60 total 80.05 last 90.88) - open/close 8086 (time 1758737618.61 total 90.05 last 82.45) - open/close 8758 (time 1758737628.61 total 100.05 last 67.20) - open/close 9606 (time 1758737638.62 total 110.07 last 84.71) - open/close 10000 (time 1758737643.15 total 114.60 last 86.87) - open/close 10863 (time 1758737653.16 total 124.61 last 86.25) - open/close 11744 (time 1758737663.16 total 134.61 last 88.04) - open/close 12528 (time 1758737673.19 total 144.64 last 78.22) - open/close 13406 (time 1758737683.19 total 154.64 last 87.79) - open/close 14233 (time 1758737693.19 total 164.64 last 82.69) - open/close 15099 (time 1758737703.20 total 174.65 last 86.54) - open/close 15997 (time 1758737713.21 total 184.66 last 89.72) - open/close 16846 (time 1758737723.21 total 194.66 last 84.84) - open/close 17716 (time 1758737733.22 total 204.66 last 86.99) - open/close 18527 (time 1758737743.22 total 214.67 last 81.08) - open/close 19439 (time 1758737753.22 total 224.67 last 91.14) - open/close 20000 (time 1758737758.81 total 230.26 last 100.34) - open/close 20907 (time 1758737768.82 total 240.27 last 90.64) - open/close 21801 (time 1758737778.84 total 250.29 last 89.26) - open/close 22678 (time 1758737788.85 total 260.30 last 87.62) - open/close 23572 (time 1758737798.85 total 270.30 last 89.38) - open/close 24431 (time 1758737808.85 total 280.30 last 85.88) - open/close 25311 (time 1758737818.85 total 290.30 last 87.99) - open/close 26188 (time 1758737828.86 total 300.31 last 87.68) - open/close 27080 (time 1758737838.86 total 310.31 last 89.15) - open/close 27974 (time 1758737848.87 total 320.32 last 89.36) - open/close 28829 (time 1758737858.87 total 330.32 last 85.44) - open/close 29657 (time 1758737868.88 total 340.33 last 82.74) - open/close 30000 (time 1758737873.16 total 344.61 last 80.10) - open/close 30889 (time 1758737883.16 total 354.61 last 88.89) - open/close 31731 (time 1758737893.16 total 364.61 last 84.19) - open/close 32669 (time 1758737903.17 total 374.62 last 93.75) - open/close 33509 (time 1758737913.17 total 384.62 last 83.96) - open/close 34391 (time 1758737923.18 total 394.63 last 88.18) - open/close 35342 (time 1758737933.18 total 404.63 last 95.09) - open/close 36305 (time 1758737943.19 total 414.64 last 96.20) - open/close 37235 (time 1758737953.20 total 424.65 last 92.89) - open/close 38198 (time 1758737963.23 total 434.68 last 96.05) - open/close 39126 (time 1758737973.23 total 444.68 last 92.74) - open/close 40000 (time 1758737982.35 total 453.80 last 95.88) - open/close 40964 (time 1758737992.35 total 463.80 last 96.37) - open/close 41945 (time 1758738002.36 total 473.81 last 98.03) - open/close 42924 (time 1758738012.36 total 483.81 last 97.85) - open/close 43885 (time 1758738022.37 total 493.82 last 96.05) - open/close 44804 (time 1758738032.38 total 503.83 last 91.80) - open/close 45768 (time 1758738042.38 total 513.83 last 96.37) - open/close 46716 (time 1758738052.39 total 523.84 last 94.74) - open/close 47644 (time 1758738062.40 total 533.85 last 92.69) - open/close 48571 (time 1758738072.40 total 543.85 last 92.68) - open/close 49494 (time 1758738082.41 total 553.85 last 92.27) - open/close 50000 (time 1758738088.55 total 560.00 last 82.30) - open/close 50796 (time 1758738098.56 total 570.00 last 79.59) - open/close 51638 (time 1758738108.58 total 580.03 last 83.98) - open/close 52546 (time 1758738118.58 total 590.03 last 90.79) - open/close 53448 (time 1758738128.59 total 600.04 last 90.15) - open/close 54304 (time 1758738138.59 total 610.04 last 85.55) - open/close 55149 (time 1758738148.60 total 620.05 last 84.45) - open/close 56038 (time 1758738158.61 total 630.06 last 88.83) - open/close 56906 (time 1758738168.62 total 640.07 last 86.70) - open/close 57818 (time 1758738178.63 total 650.08 last 91.10) - open/close 58723 (time 1758738188.64 total 660.09 last 90.41) - open/close 59616 (time 1758738198.64 total 670.09 last 89.29) - open/close 60000 (time 1758738202.84 total 674.29 last 91.51) - open/close 60865 (time 1758738212.84 total 684.29 last 86.49) - open/close 61706 (time 1758738222.85 total 694.30 last 84.05) - open/close 62605 (time 1758738232.85 total 704.30 last 89.89) - open/close 63548 (time 1758738242.85 total 714.30 last 94.23) - open/close 64527 (time 1758738252.86 total 724.31 last 97.81) total: 64768 open/close in 726.66 seconds: 89.13 ops/second - /unlink 864 (time 1758738268.03 total 10.00 last 86.38) - /unlink 1725 (time 1758738278.04 total 20.01 last 86.02) - /unlink 2579 (time 1758738288.05 total 30.02 last 85.34) - /unlink 3426 (time 1758738298.05 total 40.02 last 84.69) - /unlink 4335 (time 1758738308.06 total 50.03 last 90.83) - /unlink 5322 (time 1758738318.06 total 60.03 last 98.67) - /unlink 6383 (time 1758738328.07 total 70.04 last 106.03) - /unlink 7558 (time 1758738338.07 total 80.04 last 117.48) - /unlink 8446 (time 1758738348.07 total 90.04 last 88.78) - /unlink 9403 (time 1758738358.08 total 100.04 last 95.67) - /unlink 10000 (time 1758738365.26 total 107.23 last 83.05) - /unlink 10888 (time 1758738375.27 total 117.23 last 88.79) - /unlink 11735 (time 1758738385.27 total 127.24 last 84.67) - /unlink 12689 (time 1758738395.27 total 137.24 last 95.36) - /unlink 13634 (time 1758738405.28 total 147.25 last 94.48) - /unlink 14617 (time 1758738415.28 total 157.25 last 98.27) - /unlink 15593 (time 1758738425.28 total 167.25 last 97.57) - /unlink 16528 (time 1758738435.29 total 177.26 last 93.46) - /unlink 17420 (time 1758738445.29 total 187.26 last 89.14) - /unlink 18366 (time 1758738455.30 total 197.27 last 94.54) - /unlink 19280 (time 1758738465.31 total 207.28 last 91.35) - /unlink 20000 (time 1758738472.91 total 214.87 last 94.75) - /unlink 20863 (time 1758738482.91 total 224.88 last 86.28) - /unlink 21775 (time 1758738492.91 total 234.88 last 91.19) - /unlink 22763 (time 1758738502.91 total 244.88 last 98.77) - /unlink 23636 (time 1758738512.91 total 254.88 last 87.28) - /unlink 24599 (time 1758738522.92 total 264.89 last 96.21) - /unlink 25568 (time 1758738532.93 total 274.90 last 96.87) - /unlink 26459 (time 1758738542.93 total 284.90 last 89.04) - /unlink 27314 (time 1758738552.94 total 294.91 last 85.45) - /unlink 28097 (time 1758738562.94 total 304.91 last 78.27) - /unlink 28971 (time 1758738572.95 total 314.92 last 87.30) - /unlink 29861 (time 1758738582.96 total 324.92 last 88.98) - /unlink 30773 (time 1758738592.96 total 334.93 last 91.19) - /unlink 31706 (time 1758738602.96 total 344.93 last 93.27) - /unlink 32588 (time 1758738612.97 total 354.93 last 88.15) - /unlink 33513 (time 1758738622.97 total 364.94 last 92.50) - /unlink 34381 (time 1758738632.97 total 374.94 last 86.80) - /unlink 35242 (time 1758738642.98 total 384.94 last 86.03) - /unlink 36093 (time 1758738652.98 total 394.95 last 85.06) - /unlink 36992 (time 1758738662.99 total 404.96 last 89.82) - /unlink 37972 (time 1758738672.99 total 414.96 last 97.96) - /unlink 38896 (time 1758738683.00 total 424.97 last 92.36) - /unlink 39821 (time 1758738693.01 total 434.98 last 92.38) - /unlink 40748 (time 1758738703.02 total 444.98 last 92.67) - /unlink 41663 (time 1758738713.02 total 454.99 last 91.42) - /unlink 42605 (time 1758738723.03 total 465.00 last 94.17) - /unlink 43518 (time 1758738733.03 total 475.00 last 91.24) - /unlink 44410 (time 1758738743.05 total 485.02 last 89.04) - /unlink 45267 (time 1758738753.05 total 495.02 last 85.69) - /unlink 46166 (time 1758738763.06 total 505.02 last 89.87) - /unlink 47052 (time 1758738773.06 total 515.03 last 88.53) - /unlink 47941 (time 1758738783.07 total 525.04 last 88.86) - /unlink 48852 (time 1758738793.07 total 535.04 last 91.09) - /unlink 49744 (time 1758738803.07 total 545.04 last 89.20) - /unlink 50000 (time 1758738806.20 total 548.17 last 81.78) - /unlink 50900 (time 1758738816.20 total 558.17 last 89.96) - /unlink 51812 (time 1758738826.21 total 568.18 last 91.16) - /unlink 52742 (time 1758738836.21 total 578.18 last 92.99) - /unlink 53588 (time 1758738846.22 total 588.19 last 84.50) - /unlink 54490 (time 1758738856.23 total 598.20 last 90.16) - /unlink 55365 (time 1758738866.23 total 608.20 last 87.45) - /unlink 56323 (time 1758738876.23 total 618.20 last 95.80) - /unlink 57226 (time 1758738886.23 total 628.20 last 90.30) - /unlink 58125 (time 1758738896.23 total 638.20 last 89.89) - /unlink 58981 (time 1758738906.23 total 648.20 last 85.60) - /unlink 59861 (time 1758738916.24 total 658.21 last 87.95) - /unlink 60731 (time 1758738926.24 total 668.21 last 86.98) - /unlink 61681 (time 1758738936.24 total 678.21 last 94.98) - /unlink 62536 (time 1758738946.24 total 688.21 last 85.50) - /unlink 63537 (time 1758738956.24 total 698.21 last 100.09) - /unlink 64474 (time 1758738966.25 total 708.22 last 93.68) total: 64768 /unlink in 711.12 seconds: 91.08 ops/second Waiting for MDT destroys to complete oleg218-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 - open/close 928 (time 1758738993.48 total 10.01 last 92.75) - open/close 1746 (time 1758739003.49 total 20.01 last 81.74) - open/close 2575 (time 1758739013.49 total 30.02 last 82.87) - open/close 3382 (time 1758739023.51 total 40.03 last 80.60) - open/close 4171 (time 1758739033.51 total 50.04 last 78.84) - open/close 5020 (time 1758739043.52 total 60.05 last 84.82) - open/close 5860 (time 1758739053.53 total 70.05 last 83.94) - open/close 6735 (time 1758739063.53 total 80.05 last 87.48) - open/close 7658 (time 1758739073.54 total 90.06 last 92.22) - open/close 8437 (time 1758739083.55 total 100.07 last 77.85) - open/close 9312 (time 1758739093.55 total 110.07 last 87.47) - open/close 10000 (time 1758739102.04 total 118.56 last 81.05) - open/close 10828 (time 1758739112.04 total 128.57 last 82.76) - open/close 11762 (time 1758739122.05 total 138.57 last 93.35) - open/close 12622 (time 1758739132.05 total 148.57 last 85.97) - open/close 13509 (time 1758739142.06 total 158.59 last 88.61) - open/close 14361 (time 1758739152.07 total 168.59 last 85.18) - open/close 15202 (time 1758739162.08 total 178.60 last 83.98) - open/close 15856 (time 1758739172.10 total 188.63 last 65.24) - open/close 16498 (time 1758739182.11 total 198.63 last 64.16) - open/close 17282 (time 1758739192.11 total 208.64 last 78.36) - open/close 18094 (time 1758739202.12 total 218.64 last 81.19) - open/close 18846 (time 1758739212.13 total 228.65 last 75.08) - open/close 19646 (time 1758739222.13 total 238.66 last 79.99) - open/close 20000 (time 1758739226.14 total 242.66 last 88.35) - open/close 20780 (time 1758739236.14 total 252.66 last 78.00) - open/close 21561 (time 1758739246.15 total 262.67 last 78.04) - open/close 22427 (time 1758739256.15 total 272.67 last 86.56) - open/close 23257 (time 1758739266.16 total 282.68 last 82.97) - open/close 24182 (time 1758739276.16 total 292.68 last 92.49) - open/close 25057 (time 1758739286.16 total 302.69 last 87.45) - open/close 25955 (time 1758739296.17 total 312.70 last 89.70) - open/close 26812 (time 1758739306.18 total 322.70 last 85.67) - open/close 27634 (time 1758739316.18 total 332.70 last 82.18) - open/close 28515 (time 1758739326.19 total 342.71 last 88.04) - open/close 29384 (time 1758739336.20 total 352.72 last 86.81) - open/close 30000 (time 1758739342.99 total 359.51 last 90.69) - open/close 30908 (time 1758739352.99 total 369.52 last 90.76) - open/close 31825 (time 1758739363.00 total 379.52 last 91.68) - open/close 32667 (time 1758739373.00 total 389.52 last 84.15) - open/close 33499 (time 1758739383.00 total 399.53 last 83.18) - open/close 34315 (time 1758739393.02 total 409.55 last 81.45) - open/close 35195 (time 1758739403.03 total 419.55 last 87.94) - open/close 36116 (time 1758739413.08 total 429.60 last 91.66) - open/close 37026 (time 1758739423.09 total 439.61 last 90.90) - open/close 37922 (time 1758739433.10 total 449.62 last 89.48) - open/close 38716 (time 1758739443.10 total 459.62 last 79.40) - open/close 39493 (time 1758739453.11 total 469.63 last 77.66) - open/close 40000 (time 1758739460.67 total 477.19 last 67.02) - open/close 40739 (time 1758739470.67 total 487.19 last 73.89) - open/close 41691 (time 1758739480.67 total 497.20 last 95.18) - open/close 42547 (time 1758739490.68 total 507.20 last 85.54) - open/close 43376 (time 1758739500.70 total 517.22 last 82.76) - open/close 44257 (time 1758739510.70 total 527.23 last 88.05) - open/close 45115 (time 1758739520.71 total 537.23 last 85.77) - open/close 45935 (time 1758739530.71 total 547.24 last 81.96) - open/close 46798 (time 1758739540.71 total 557.24 last 86.29) - open/close 47627 (time 1758739550.72 total 567.25 last 82.83) - open/close 48469 (time 1758739560.74 total 577.26 last 84.04) - open/close 49170 (time 1758739570.75 total 587.27 last 70.05) - open/close 49999 (time 1758739580.75 total 597.27 last 82.88) - open/close 50881 (time 1758739590.75 total 607.28 last 88.17) - open/close 51642 (time 1758739600.76 total 617.28 last 76.07) - open/close 52321 (time 1758739610.77 total 627.29 last 67.83) - open/close 53076 (time 1758739620.81 total 637.33 last 75.22) - open/close 53790 (time 1758739630.81 total 647.33 last 71.39) - open/close 54536 (time 1758739640.82 total 657.34 last 74.55) - open/close 55352 (time 1758739650.83 total 667.35 last 81.50) - open/close 56232 (time 1758739660.83 total 677.35 last 87.97) - open/close 57079 (time 1758739670.84 total 687.36 last 84.66) - open/close 57902 (time 1758739680.85 total 697.37 last 82.19) - open/close 58755 (time 1758739690.87 total 707.39 last 85.17) - open/close 59592 (time 1758739700.89 total 717.41 last 83.52) - open/close 60000 (time 1758739705.37 total 721.89 last 90.98) - open/close 60860 (time 1758739715.38 total 731.90 last 85.90) - open/close 61660 (time 1758739725.39 total 741.91 last 79.96) - open/close 62461 (time 1758739735.39 total 751.91 last 80.07) - open/close 63297 (time 1758739745.39 total 761.92 last 83.57) - open/close 64178 (time 1758739755.40 total 771.92 last 88.07) total: 64768 open/close in 778.35 seconds: 83.21 ops/second - /unlink 900 (time 1758739775.69 total 10.01 last 89.95) - /unlink 1809 (time 1758739785.69 total 20.01 last 90.90) - /unlink 2683 (time 1758739795.69 total 30.01 last 87.38) - /unlink 3594 (time 1758739805.70 total 40.02 last 91.01) - /unlink 4467 (time 1758739815.71 total 50.03 last 87.24) - /unlink 5342 (time 1758739825.71 total 60.03 last 87.48) - /unlink 6214 (time 1758739835.72 total 70.03 last 87.15) - /unlink 7096 (time 1758739845.72 total 80.04 last 88.17) - /unlink 7938 (time 1758739855.72 total 90.04 last 84.17) - /unlink 8766 (time 1758739865.74 total 100.05 last 82.69) - /unlink 9640 (time 1758739875.74 total 110.06 last 87.35) - /unlink 10000 (time 1758739879.94 total 114.25 last 85.82) - /unlink 10926 (time 1758739889.94 total 124.25 last 92.60) - /unlink 11783 (time 1758739899.94 total 134.25 last 85.70) - /unlink 12634 (time 1758739909.94 total 144.26 last 85.04) - /unlink 13495 (time 1758739919.95 total 154.26 last 86.08) - /unlink 14384 (time 1758739929.95 total 164.27 last 88.85) - /unlink 15274 (time 1758739939.95 total 174.27 last 88.99) - /unlink 16145 (time 1758739949.95 total 184.27 last 87.09) - /unlink 17080 (time 1758739959.96 total 194.27 last 93.48) - /unlink 17961 (time 1758739969.96 total 204.28 last 88.08) - /unlink 18817 (time 1758739979.97 total 214.29 last 85.51) - /unlink 19654 (time 1758739989.98 total 224.30 last 83.62) - /unlink 20000 (time 1758739993.96 total 228.28 last 86.92) - /unlink 20880 (time 1758740003.96 total 238.28 last 87.96) - /unlink 21762 (time 1758740013.98 total 248.29 last 88.10) - /unlink 22700 (time 1758740023.98 total 258.30 last 93.73) - /unlink 23615 (time 1758740033.99 total 268.30 last 91.48) - /unlink 24544 (time 1758740044.00 total 278.31 last 92.82) - /unlink 25429 (time 1758740054.00 total 288.32 last 88.48) - /unlink 26279 (time 1758740064.00 total 298.32 last 84.99) - /unlink 27137 (time 1758740074.01 total 308.32 last 85.75) - /unlink 28009 (time 1758740084.02 total 318.33 last 87.10) - /unlink 28868 (time 1758740094.02 total 328.33 last 85.89) - /unlink 29772 (time 1758740104.03 total 338.34 last 90.33) - /unlink 30000 (time 1758740106.56 total 340.87 last 90.04) - /unlink 30892 (time 1758740116.57 total 350.88 last 89.12) - /unlink 31774 (time 1758740126.57 total 360.89 last 88.12) - /unlink 32648 (time 1758740136.58 total 370.90 last 87.31) - /unlink 33570 (time 1758740146.59 total 380.90 last 92.17) - /unlink 34507 (time 1758740156.59 total 390.91 last 93.69) - /unlink 35437 (time 1758740166.59 total 400.91 last 92.99) - /unlink 36376 (time 1758740176.60 total 410.92 last 93.80) - /unlink 37329 (time 1758740186.61 total 420.93 last 95.23) - /unlink 38240 (time 1758740196.61 total 430.93 last 91.04) - /unlink 39162 (time 1758740206.61 total 440.93 last 92.20) - /unlink 40000 (time 1758740216.31 total 450.63 last 86.45) - /unlink 40914 (time 1758740226.32 total 460.64 last 91.29) - /unlink 41856 (time 1758740236.33 total 470.64 last 94.14) - /unlink 42792 (time 1758740246.34 total 480.66 last 93.47) - /unlink 43747 (time 1758740256.34 total 490.66 last 95.49) - /unlink 44624 (time 1758740266.34 total 500.66 last 87.67) - /unlink 45509 (time 1758740276.35 total 510.67 last 88.45) - /unlink 46427 (time 1758740286.35 total 520.67 last 91.77) - /unlink 47354 (time 1758740296.36 total 530.68 last 92.66) - /unlink 48305 (time 1758740306.36 total 540.68 last 95.08) - /unlink 49209 (time 1758740316.36 total 550.68 last 90.38) - /unlink 50000 (time 1758740324.63 total 558.95 last 95.67) - /unlink 50973 (time 1758740334.64 total 568.96 last 97.21) - /unlink 51913 (time 1758740344.65 total 578.97 last 93.92) - /unlink 52840 (time 1758740354.65 total 588.97 last 92.68) - /unlink 53757 (time 1758740364.65 total 598.97 last 91.69) - /unlink 54706 (time 1758740374.66 total 608.97 last 94.88) - /unlink 55636 (time 1758740384.66 total 618.98 last 92.98) - /unlink 56559 (time 1758740394.67 total 628.99 last 92.15) - /unlink 57493 (time 1758740404.68 total 639.00 last 93.37) - /unlink 58440 (time 1758740414.68 total 649.00 last 94.64) - /unlink 59388 (time 1758740424.69 total 659.01 last 94.72) - /unlink 60000 (time 1758740431.41 total 665.72 last 91.14) - /unlink 60928 (time 1758740441.41 total 675.73 last 92.73) - /unlink 61895 (time 1758740451.42 total 685.74 last 96.65) - /unlink 62845 (time 1758740461.42 total 695.74 last 94.99) - /unlink 63774 (time 1758740471.42 total 705.74 last 92.90) - /unlink 64725 (time 1758740481.43 total 715.74 last 95.06) total: 64768 /unlink in 716.11 seconds: 90.44 ops/second Waiting for MDT destroys to complete oleg218-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 - open/close 968 (time 1758740508.16 total 10.00 last 96.77) - open/close 1903 (time 1758740518.17 total 20.01 last 93.41) - open/close 2828 (time 1758740528.18 total 30.02 last 92.44) - open/close 3870 (time 1758740538.18 total 40.02 last 104.17) - open/close 4851 (time 1758740548.19 total 50.03 last 98.02) - open/close 5782 (time 1758740558.20 total 60.04 last 93.01) - open/close 6814 (time 1758740568.20 total 70.04 last 103.16) - open/close 7739 (time 1758740578.21 total 80.05 last 92.44) - open/close 8685 (time 1758740588.22 total 90.06 last 94.55) - open/close 9628 (time 1758740598.22 total 100.06 last 94.26) - open/close 10000 (time 1758740602.03 total 103.87 last 97.60) - open/close 10914 (time 1758740612.03 total 113.87 last 91.39) - open/close 11834 (time 1758740622.04 total 123.88 last 91.97) - open/close 12761 (time 1758740632.04 total 133.88 last 92.64) - open/close 13690 (time 1758740642.04 total 143.88 last 92.89) - open/close 14620 (time 1758740652.05 total 153.89 last 92.93) - open/close 15600 (time 1758740662.06 total 163.90 last 97.94) - open/close 16611 (time 1758740672.06 total 173.90 last 101.09) - open/close 17650 (time 1758740682.06 total 183.90 last 103.85) - open/close 18550 (time 1758740692.07 total 193.91 last 89.94) - open/close 19442 (time 1758740702.08 total 203.92 last 89.14) - open/close 20000 (time 1758740708.14 total 209.98 last 92.04) - open/close 21019 (time 1758740718.15 total 219.99 last 101.74) - open/close 22037 (time 1758740728.16 total 230.00 last 101.77) - open/close 23064 (time 1758740738.16 total 240.00 last 102.65) - open/close 24153 (time 1758740748.16 total 250.00 last 108.86) - open/close 25114 (time 1758740758.17 total 260.01 last 96.08) - open/close 26113 (time 1758740768.17 total 270.01 last 99.89) - open/close 27166 (time 1758740778.17 total 280.01 last 105.28) - open/close 28255 (time 1758740788.17 total 290.01 last 108.88) - open/close 29255 (time 1758740798.18 total 300.02 last 99.94) - open/close 30000 (time 1758740805.76 total 307.60 last 98.29) - open/close 31083 (time 1758740815.76 total 317.60 last 108.23) - open/close 32188 (time 1758740825.77 total 327.61 last 110.46) - open/close 33230 (time 1758740835.77 total 337.61 last 104.15) - open/close 34333 (time 1758740845.77 total 347.61 last 110.27) - open/close 35393 (time 1758740855.78 total 357.62 last 105.91) - open/close 36554 (time 1758740865.79 total 367.63 last 116.06) - open/close 37726 (time 1758740875.79 total 377.63 last 117.19) - open/close 38749 (time 1758740885.80 total 387.64 last 102.19) - open/close 39808 (time 1758740895.81 total 397.65 last 105.81) - open/close 40905 (time 1758740905.81 total 407.65 last 109.67) - open/close 41972 (time 1758740915.81 total 417.65 last 106.64) - open/close 43168 (time 1758740925.82 total 427.66 last 119.55) - open/close 44387 (time 1758740935.82 total 437.66 last 121.81) - open/close 45557 (time 1758740945.83 total 447.67 last 116.97) - open/close 46834 (time 1758740955.83 total 457.67 last 127.64) - open/close 47993 (time 1758740965.84 total 467.68 last 115.85) - open/close 49248 (time 1758740975.84 total 477.68 last 125.46) - open/close 50000 (time 1758740981.54 total 483.38 last 131.91) - open/close 51362 (time 1758740991.54 total 493.38 last 136.14) - open/close 52860 (time 1758741001.55 total 503.39 last 149.72) - open/close 54235 (time 1758741011.56 total 513.40 last 137.43) - open/close 55464 (time 1758741021.56 total 523.40 last 122.85) - open/close 56746 (time 1758741031.57 total 533.41 last 128.11) - open/close 58087 (time 1758741041.57 total 543.41 last 134.02) - open/close 59691 (time 1758741051.58 total 553.42 last 160.32) - open/close 60000 (time 1758741053.77 total 555.61 last 140.83) - open/close 61284 (time 1758741063.78 total 565.62 last 128.34) - open/close 62860 (time 1758741073.78 total 575.62 last 157.50) - open/close 64309 (time 1758741083.79 total 585.63 last 144.79) total: 64768 open/close in 589.42 seconds: 109.88 ops/second - /unlink 2448 (time 1758741099.84 total 10.00 last 244.76) - /unlink 4985 (time 1758741109.84 total 20.00 last 253.68) - /unlink 7354 (time 1758741119.85 total 30.01 last 236.83) - /unlink 9432 (time 1758741129.85 total 40.01 last 207.79) - /unlink 10000 (time 1758741133.19 total 43.35 last 169.82) - /unlink 11831 (time 1758741143.19 total 53.35 last 183.09) - /unlink 14018 (time 1758741153.20 total 63.35 last 218.63) - /unlink 16331 (time 1758741163.20 total 73.36 last 231.26) - /unlink 18103 (time 1758741173.20 total 83.36 last 177.13) - /unlink 19712 (time 1758741183.20 total 93.36 last 160.90) - /unlink 21600 (time 1758741193.21 total 103.36 last 188.72) - /unlink 23710 (time 1758741203.21 total 113.37 last 210.90) - /unlink 26141 (time 1758741213.21 total 123.37 last 243.06) - /unlink 28303 (time 1758741223.22 total 133.37 last 216.12) - /unlink 30000 (time 1758741230.20 total 140.36 last 243.04) - /unlink 33114 (time 1758741240.20 total 150.36 last 311.27) - /unlink 36257 (time 1758741250.20 total 160.36 last 314.29) - /unlink 38915 (time 1758741260.21 total 170.37 last 265.70) - /unlink 40000 (time 1758741264.51 total 174.67 last 252.00) - /unlink 42839 (time 1758741274.52 total 184.67 last 283.82) - /unlink 45720 (time 1758741284.52 total 194.68 last 288.04) - /unlink 48738 (time 1758741294.52 total 204.68 last 301.76) - /unlink 50000 (time 1758741299.77 total 209.93 last 240.28) - /unlink 53051 (time 1758741309.77 total 219.93 last 305.08) - /unlink 56323 (time 1758741319.77 total 229.93 last 327.17) - /unlink 59784 (time 1758741329.77 total 239.93 last 346.07) - /unlink 63451 (time 1758741339.78 total 249.93 last 366.61) total: 64768 /unlink in 252.90 seconds: 256.10 ops/second Waiting for MDT destroys to complete oleg218-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 - open/close 2251 (time 1758741365.83 total 10.00 last 225.00) - open/close 4644 (time 1758741375.84 total 20.01 last 239.07) - open/close 6411 (time 1758741385.85 total 30.02 last 176.55) - open/close 8336 (time 1758741395.86 total 40.03 last 192.43) - open/close 9978 (time 1758741405.86 total 50.03 last 164.06) - open/close 11495 (time 1758741415.87 total 60.04 last 151.63) - open/close 13437 (time 1758741425.87 total 70.04 last 194.09) - open/close 15419 (time 1758741435.88 total 80.05 last 198.12) - open/close 17668 (time 1758741445.88 total 90.05 last 224.83) - open/close 20000 (time 1758741454.55 total 98.72 last 268.87) - open/close 22209 (time 1758741464.56 total 108.73 last 220.85) - open/close 24570 (time 1758741474.56 total 118.73 last 236.02) - open/close 26352 (time 1758741484.57 total 128.74 last 178.07) - open/close 28052 (time 1758741494.57 total 138.74 last 169.98) - open/close 29894 (time 1758741504.57 total 148.74 last 184.15) - open/close 31483 (time 1758741514.58 total 158.75 last 158.81) - open/close 33155 (time 1758741524.58 total 168.75 last 167.15) - open/close 35155 (time 1758741534.58 total 178.75 last 199.99) - open/close 37059 (time 1758741544.59 total 188.76 last 190.30) - open/close 39105 (time 1758741554.59 total 198.76 last 204.50) - open/close 40000 (time 1758741558.75 total 202.92 last 215.38) - open/close 42147 (time 1758741568.75 total 212.92 last 214.65) - open/close 44553 (time 1758741578.75 total 222.93 last 240.44) - open/close 46718 (time 1758741588.76 total 232.93 last 216.47) - open/close 48878 (time 1758741598.76 total 242.93 last 215.96) - open/close 50000 (time 1758741603.59 total 247.76 last 232.18) - open/close 52455 (time 1758741613.59 total 257.76 last 245.45) - open/close 55011 (time 1758741623.59 total 267.76 last 255.59) - open/close 57708 (time 1758741633.60 total 277.77 last 269.61) - open/close 60000 (time 1758741643.20 total 287.37 last 238.75) - open/close 62998 (time 1758741653.20 total 297.37 last 299.71) total: 64768 open/close in 303.37 seconds: 213.50 ops/second - /unlink 6128 (time 1758741670.70 total 10.00 last 612.77) - /unlink 10000 (time 1758741676.42 total 15.72 last 676.74) - /unlink 16154 (time 1758741686.42 total 25.72 last 615.37) - /unlink 20000 (time 1758741692.24 total 31.54 last 661.15) - /unlink 27037 (time 1758741702.24 total 41.54 last 703.67) - /unlink 30000 (time 1758741706.42 total 45.73 last 707.79) - /unlink 36361 (time 1758741716.42 total 55.73 last 636.09) - /unlink 40000 (time 1758741722.57 total 61.87 last 591.85) - /unlink 46028 (time 1758741732.57 total 71.88 last 602.74) - /unlink 50000 (time 1758741737.93 total 77.23 last 741.87) - /unlink 56808 (time 1758741747.93 total 87.23 last 680.79) - /unlink 60000 (time 1758741751.92 total 91.22 last 799.68) total: 64768 /unlink in 98.21 seconds: 659.47 ops/second Waiting for MDT destroys to complete oleg218-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 - open/close 4495 (time 1758741816.28 total 10.00 last 449.44) - open/close 8025 (time 1758741826.28 total 20.00 last 352.93) - open/close 10000 (time 1758741830.95 total 24.67 last 422.99) - open/close 13685 (time 1758741840.95 total 34.67 last 368.49) - open/close 17331 (time 1758741850.95 total 44.68 last 364.48) - open/close 20000 (time 1758741858.45 total 52.17 last 356.11) - open/close 23731 (time 1758741868.45 total 62.17 last 373.05) - open/close 28345 (time 1758741878.45 total 72.17 last 461.34) - open/close 30000 (time 1758741882.63 total 76.35 last 396.43) - open/close 33970 (time 1758741892.63 total 86.35 last 396.98) - open/close 37651 (time 1758741902.63 total 96.35 last 368.04) - open/close 40000 (time 1758741908.81 total 102.53 last 380.05) - open/close 43322 (time 1758741918.81 total 112.53 last 332.14) - open/close 46120 (time 1758741928.81 total 122.53 last 279.76) - open/close 48985 (time 1758741938.81 total 132.54 last 286.47) - open/close 50000 (time 1758741943.10 total 136.82 last 236.65) - open/close 52939 (time 1758741953.10 total 146.83 last 293.88) - open/close 56196 (time 1758741963.10 total 156.83 last 325.63) - open/close 58937 (time 1758741973.11 total 166.83 last 274.03) - open/close 60000 (time 1758741977.08 total 170.80 last 267.64) - open/close 63160 (time 1758741987.08 total 180.80 last 315.94) total: 64768 open/close in 185.64 seconds: 348.88 ops/second - /unlink 6298 (time 1758742003.50 total 10.00 last 629.76) - /unlink 10000 (time 1758742009.54 total 16.04 last 612.60) - /unlink 16912 (time 1758742019.54 total 26.04 last 691.13) - /unlink 20000 (time 1758742024.02 total 30.52 last 690.10) - /unlink 26735 (time 1758742034.02 total 40.52 last 673.36) - /unlink 30000 (time 1758742038.45 total 44.96 last 735.75) - /unlink 37179 (time 1758742048.46 total 54.96 last 717.88) - /unlink 40000 (time 1758742052.60 total 59.11 last 680.45) - /unlink 47844 (time 1758742062.60 total 69.11 last 784.39) - /unlink 50000 (time 1758742065.50 total 72.00 last 744.18) - /unlink 55957 (time 1758742075.50 total 82.00 last 595.62) - /unlink 60000 (time 1758742083.11 total 89.61 last 531.45) total: 64768 /unlink in 98.49 seconds: 657.62 ops/second Waiting for MDT destroys to complete oleg218-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2