== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:01:29 (1787785289) start mds service on oleg351-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg351-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1787785376.66 total 10.01 last 24.38) - open/close 433 (time 1787785386.82 total 20.17 last 18.60) - open/close 617 (time 1787785397.23 total 30.57 last 17.69) - open/close 765 (time 1787785407.56 total 40.90 last 14.33) - open/close 968 (time 1787785417.57 total 50.91 last 20.28) - open/close 1155 (time 1787785428.42 total 61.77 last 17.23) - open/close 1388 (time 1787785439.04 total 72.39 last 21.94) - open/close 1607 (time 1787785449.46 total 82.81 last 21.02) - open/close 1744 (time 1787785459.47 total 92.81 last 13.70) - open/close 1981 (time 1787785469.48 total 102.82 last 23.68) - open/close 2202 (time 1787785480.26 total 113.60 last 20.50) - open/close 2405 (time 1787785490.68 total 124.03 last 19.47) - open/close 2579 (time 1787785500.70 total 134.04 last 17.38) - open/close 2704 (time 1787785511.76 total 145.10 last 11.30) - open/close 2844 (time 1787785522.27 total 155.61 last 13.32) - open/close 2968 (time 1787785533.30 total 166.65 last 11.24) - open/close 3191 (time 1787785544.08 total 177.42 last 20.69) - open/close 3350 (time 1787785554.56 total 187.90 last 15.17) - open/close 3596 (time 1787785564.58 total 197.92 last 24.56) - open/close 3795 (time 1787785574.58 total 207.93 last 19.89) - open/close 3944 (time 1787785584.60 total 217.95 last 14.87) - open/close 4158 (time 1787785595.12 total 228.47 last 20.34) - open/close 4333 (time 1787785605.50 total 238.85 last 16.86) - open/close 4562 (time 1787785615.64 total 248.98 last 22.60) - open/close 4762 (time 1787785626.56 total 259.91 last 18.30) - open/close 4923 (time 1787785637.37 total 270.71 last 14.91) - open/close 5059 (time 1787785647.74 total 281.08 last 13.11) - open/close 5268 (time 1787785658.69 total 292.03 last 19.09) - open/close 5391 (time 1787785668.70 total 302.04 last 12.29) - open/close 5534 (time 1787785678.71 total 312.05 last 14.28) - open/close 5710 (time 1787785689.46 total 322.80 last 16.37) - open/close 5926 (time 1787785699.96 total 333.31 last 20.57) - open/close 6149 (time 1787785710.12 total 343.47 last 21.94) - open/close 6434 (time 1787785721.19 total 354.54 last 25.75) - open/close 6638 (time 1787785731.33 total 364.67 last 20.13) - open/close 6848 (time 1787785741.84 total 375.18 last 19.98) - open/close 7000 (time 1787785752.11 total 385.46 last 14.80) - open/close 7240 (time 1787785762.12 total 395.46 last 23.99) - open/close 7499 (time 1787785772.78 total 406.12 last 24.29) - open/close 7741 (time 1787785783.74 total 417.09 last 22.07) - open/close 7921 (time 1787785793.75 total 427.10 last 17.98) - open/close 8128 (time 1787785804.47 total 437.81 last 19.31) - open/close 8291 (time 1787785815.36 total 448.71 last 14.96) - open/close 8549 (time 1787785825.36 total 458.71 last 25.79) - open/close 8698 (time 1787785835.37 total 468.71 last 14.90) - open/close 8927 (time 1787785846.77 total 480.12 last 20.08) - open/close 9144 (time 1787785857.41 total 490.76 last 20.39) - open/close 9392 (time 1787785867.44 total 500.78 last 24.74) - open/close 9596 (time 1787785877.45 total 510.79 last 20.38) - open/close 9737 (time 1787785888.44 total 521.79 last 12.82) - open/close 9819 (time 1787785898.46 total 531.80 last 8.19) - open/close 10000 (time 1787785906.25 total 539.60 last 23.21) - open/close 10137 (time 1787785916.33 total 549.68 last 13.59) - open/close 10309 (time 1787785927.35 total 560.70 last 15.61) - open/close 10567 (time 1787785938.36 total 571.70 last 23.44) - open/close 10804 (time 1787785948.37 total 581.72 last 23.67) - open/close 11031 (time 1787785959.03 total 592.38 last 21.29) - open/close 11229 (time 1787785969.14 total 602.49 last 19.58) - open/close 11476 (time 1787785979.15 total 612.50 last 24.68) - open/close 11659 (time 1787785989.18 total 622.53 last 18.25) - open/close 11855 (time 1787785999.18 total 632.53 last 19.59) - open/close 12050 (time 1787786009.28 total 642.62 last 19.32) - open/close 12289 (time 1787786019.28 total 652.63 last 23.89) - open/close 12496 (time 1787786029.61 total 662.96 last 20.04) - open/close 12747 (time 1787786040.18 total 673.52 last 23.76) - open/close 12975 (time 1787786050.99 total 684.34 last 21.08) - open/close 13213 (time 1787786061.23 total 694.57 last 23.25) - open/close 13465 (time 1787786071.26 total 704.61 last 25.11) - open/close 13712 (time 1787786081.30 total 714.65 last 24.61) - open/close 14004 (time 1787786091.33 total 724.68 last 29.12) - open/close 14223 (time 1787786101.36 total 734.71 last 21.83) - open/close 14444 (time 1787786111.96 total 745.30 last 20.86) - open/close 14696 (time 1787786122.87 total 756.21 last 23.10) - open/close 14901 (time 1787786132.87 total 766.22 last 20.49) - open/close 15159 (time 1787786143.56 total 776.91 last 24.13) - open/close 15379 (time 1787786153.66 total 787.01 last 21.77) - open/close 15612 (time 1787786164.16 total 797.50 last 22.20) - open/close 15843 (time 1787786174.26 total 807.60 last 22.88) - open/close 16099 (time 1787786184.81 total 818.16 last 24.25) - open/close 16321 (time 1787786195.59 total 828.94 last 20.59) - open/close 16567 (time 1787786206.46 total 839.80 last 22.64) - open/close 16809 (time 1787786217.46 total 850.80 last 22.01) - open/close 17058 (time 1787786228.01 total 861.35 last 23.59) - open/close 17251 (time 1787786238.96 total 872.30 last 17.63) - open/close 17366 (time 1787786248.98 total 882.33 last 11.47) - open/close 17532 (time 1787786259.96 total 893.30 last 15.13) - open/close 17734 (time 1787786270.26 total 903.60 last 19.61) - open/close 18028 (time 1787786280.97 total 914.31 last 27.45) - open/close 18239 (time 1787786290.98 total 924.32 last 21.08) - open/close 18486 (time 1787786300.98 total 934.33 last 24.69) - open/close 18690 (time 1787786311.04 total 944.38 last 20.29) - open/close 19168 (time 1787786321.05 total 954.39 last 47.75) - open/close 19882 (time 1787786331.06 total 964.40 last 71.34) - open/close 20534 (time 1787786341.06 total 974.41 last 65.17) - open/close 21223 (time 1787786351.08 total 984.42 last 68.80) - open/close 21902 (time 1787786361.08 total 994.43 last 67.87) - open/close 22593 (time 1787786371.09 total 1004.43 last 69.06) - open/close 23230 (time 1787786381.09 total 1014.44 last 63.68) - open/close 23952 (time 1787786391.09 total 1024.44 last 72.16) - open/close 24577 (time 1787786401.12 total 1034.47 last 62.33) - open/close 25274 (time 1787786411.13 total 1044.47 last 69.64) - open/close 25976 (time 1787786421.13 total 1054.47 last 70.20) - open/close 26654 (time 1787786431.14 total 1064.48 last 67.76) - open/close 27298 (time 1787786441.15 total 1074.49 last 64.33) - open/close 28007 (time 1787786451.16 total 1084.51 last 70.79) - open/close 28776 (time 1787786461.17 total 1094.51 last 76.85) - open/close 29467 (time 1787786471.17 total 1104.52 last 69.09) - open/close 30000 (time 1787786478.71 total 1112.06 last 70.66) - open/close 30709 (time 1787786488.73 total 1122.07 last 70.81) - open/close 31371 (time 1787786498.73 total 1132.08 last 66.16) - open/close 32034 (time 1787786508.75 total 1142.09 last 66.21) - open/close 32675 (time 1787786518.75 total 1152.10 last 64.07) - open/close 33399 (time 1787786528.75 total 1162.10 last 72.38) - open/close 34078 (time 1787786538.76 total 1172.10 last 67.89) - open/close 34784 (time 1787786548.77 total 1182.11 last 70.53) - open/close 35456 (time 1787786558.78 total 1192.12 last 67.12) - open/close 36115 (time 1787786568.78 total 1202.12 last 65.90) - open/close 36801 (time 1787786578.78 total 1212.12 last 68.60) - open/close 37461 (time 1787786588.79 total 1222.13 last 65.95) - open/close 38121 (time 1787786598.79 total 1232.14 last 65.97) - open/close 38763 (time 1787786608.81 total 1242.16 last 64.07) - open/close 39446 (time 1787786618.93 total 1252.28 last 67.47) - open/close 40000 (time 1787786628.64 total 1261.99 last 57.07) - open/close 40557 (time 1787786638.65 total 1271.99 last 55.68) - open/close 41190 (time 1787786648.65 total 1281.99 last 63.29) - open/close 41808 (time 1787786658.66 total 1292.01 last 61.71) - open/close 42490 (time 1787786668.68 total 1302.02 last 68.09) - open/close 43099 (time 1787786678.69 total 1312.04 last 60.80) - open/close 43750 (time 1787786688.71 total 1322.06 last 64.98) - open/close 44391 (time 1787786698.72 total 1332.07 last 64.02) - open/close 45035 (time 1787786708.73 total 1342.08 last 64.35) - open/close 45656 (time 1787786718.74 total 1352.09 last 62.04) - open/close 46274 (time 1787786728.76 total 1362.11 last 61.65) - open/close 46899 (time 1787786738.77 total 1372.11 last 62.48) - open/close 47495 (time 1787786748.77 total 1382.12 last 59.57) - open/close 48043 (time 1787786758.78 total 1392.13 last 54.74) - open/close 48546 (time 1787786768.79 total 1402.13 last 50.29) - open/close 49126 (time 1787786778.79 total 1412.14 last 57.96) - open/close 49732 (time 1787786788.81 total 1422.16 last 60.47) total: 50000 open/close in 1426.74 seconds: 35.05 ops/second - unlinked 0 (time 1787786797 ; total 0 ; last 0) - unlinked 10000 (time 1787786961 ; total 164 ; last 164) - unlinked 20000 (time 1787787125 ; total 328 ; last 164) - unlinked 30000 (time 1787787284 ; total 487 ; last 159) - unlinked 40000 (time 1787787439 ; total 642 ; last 155) total: 50000 unlinks in 798 seconds: 62.656643 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2