== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:20:10 (1787700010) start mds service on oleg612-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing load_modules_local oleg612-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg612-server: Loading modules from /home/green/git/lustre-release/lustre oleg612-server: detected 4 online CPUs by sysfs oleg612-server: MODOPTS_LIBCFS= oleg612-server: Force libcfs to create 2 CPU partitions oleg612-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg612-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg612-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 236 (time 1787700094.09 total 10.01 last 23.57) - open/close 401 (time 1787700104.11 total 20.03 last 16.47) - open/close 592 (time 1787700114.91 total 30.83 last 17.68) - open/close 720 (time 1787700124.91 total 40.84 last 12.79) - open/close 876 (time 1787700134.95 total 50.87 last 15.55) - open/close 1110 (time 1787700145.24 total 61.16 last 22.74) - open/close 1328 (time 1787700156.32 total 72.25 last 19.67) - open/close 1596 (time 1787700167.12 total 83.05 last 24.82) - open/close 1881 (time 1787700177.13 total 93.06 last 28.47) - open/close 2104 (time 1787700187.29 total 103.22 last 21.95) - open/close 2310 (time 1787700197.30 total 113.23 last 20.58) - open/close 2480 (time 1787700207.51 total 123.43 last 16.66) - open/close 2680 (time 1787700218.13 total 134.06 last 18.82) - open/close 2924 (time 1787700230.32 total 146.24 last 20.02) - open/close 3129 (time 1787700240.33 total 156.25 last 20.48) - open/close 3353 (time 1787700251.27 total 167.20 last 20.46) - open/close 3551 (time 1787700261.86 total 177.78 last 18.71) - open/close 3810 (time 1787700271.86 total 187.79 last 25.89) - open/close 4046 (time 1787700282.85 total 198.77 last 21.48) - open/close 4200 (time 1787700293.41 total 209.34 last 14.58) - open/close 4409 (time 1787700303.43 total 219.35 last 20.87) - open/close 4609 (time 1787700313.45 total 229.38 last 19.95) - open/close 4784 (time 1787700323.95 total 239.87 last 16.68) - open/close 5028 (time 1787700333.96 total 249.88 last 24.38) - open/close 5229 (time 1787700343.98 total 259.90 last 20.06) - open/close 5426 (time 1787700353.99 total 269.91 last 19.68) - open/close 5660 (time 1787700364.00 total 279.92 last 23.38) - open/close 5750 (time 1787700375.26 total 291.19 last 7.99) - open/close 5928 (time 1787700385.57 total 301.49 last 17.27) - open/close 6093 (time 1787700396.07 total 311.99 last 15.71) - open/close 6182 (time 1787700406.14 total 322.06 last 8.84) - open/close 6262 (time 1787700416.67 total 332.60 last 7.59) - open/close 6337 (time 1787700426.70 total 342.62 last 7.48) - open/close 6473 (time 1787700436.96 total 352.88 last 13.25) - open/close 6556 (time 1787700447.80 total 363.73 last 7.65) - open/close 6664 (time 1787700458.67 total 374.59 last 9.94) - open/close 6757 (time 1787700469.60 total 385.53 last 8.51) - open/close 6875 (time 1787700479.76 total 395.68 last 11.62) - open/close 7066 (time 1787700489.77 total 405.70 last 19.07) - open/close 7281 (time 1787700501.60 total 417.52 last 18.18) - open/close 7387 (time 1787700512.60 total 428.52 last 9.64) - open/close 7523 (time 1787700524.06 total 439.99 last 11.86) - open/close 7718 (time 1787700534.47 total 450.40 last 18.73) - open/close 7918 (time 1787700545.31 total 461.24 last 18.45) - open/close 8063 (time 1787700556.38 total 472.30 last 13.11) - open/close 8270 (time 1787700566.86 total 482.79 last 19.74) - open/close 8507 (time 1787700576.87 total 492.80 last 23.68) - open/close 8636 (time 1787700587.17 total 503.09 last 12.53) - open/close 8859 (time 1787700597.19 total 513.12 last 22.25) - open/close 8990 (time 1787700607.20 total 523.12 last 13.09) - open/close 9169 (time 1787700618.26 total 534.19 last 16.18) - open/close 9405 (time 1787700630.49 total 546.41 last 19.31) - open/close 9651 (time 1787700641.06 total 556.98 last 23.27) - open/close 9871 (time 1787700651.38 total 567.30 last 21.31) - open/close 10000 (time 1787700659.19 total 575.11 last 16.52) - open/close 10209 (time 1787700671.38 total 587.30 last 17.14) - open/close 10290 (time 1787700681.61 total 597.54 last 7.92) - open/close 10522 (time 1787700691.63 total 607.55 last 23.16) - open/close 10743 (time 1787700701.77 total 617.70 last 21.78) - open/close 10995 (time 1787700714.78 total 630.71 last 19.37) - open/close 11173 (time 1787700725.60 total 641.53 last 16.45) - open/close 11375 (time 1787700735.72 total 651.64 last 19.97) - open/close 11571 (time 1787700746.39 total 662.32 last 18.36) - open/close 11836 (time 1787700757.13 total 673.06 last 24.67) - open/close 12093 (time 1787700767.70 total 683.62 last 24.33) - open/close 12366 (time 1787700777.71 total 693.64 last 27.26) - open/close 12617 (time 1787700787.99 total 703.92 last 24.41) - open/close 12824 (time 1787700798.79 total 714.71 last 19.18) - open/close 13021 (time 1787700808.91 total 724.84 last 19.45) - open/close 13195 (time 1787700819.29 total 735.21 last 16.77) - open/close 13405 (time 1787700829.96 total 745.89 last 19.67) - open/close 13536 (time 1787700840.00 total 755.92 last 13.06) - open/close 13755 (time 1787700850.47 total 766.40 last 20.90) - open/close 13969 (time 1787700860.48 total 776.40 last 21.39) - open/close 14150 (time 1787700870.48 total 786.41 last 18.10) - open/close 14430 (time 1787700881.38 total 797.30 last 25.70) - open/close 14693 (time 1787700891.68 total 807.60 last 25.53) - open/close 14887 (time 1787700901.71 total 817.63 last 19.34) - open/close 15101 (time 1787700911.73 total 827.66 last 21.35) - open/close 15266 (time 1787700921.73 total 837.66 last 16.50) - open/close 15502 (time 1787700931.74 total 847.67 last 23.58) - open/close 15735 (time 1787700942.82 total 858.74 last 21.04) - open/close 15962 (time 1787700952.83 total 868.75 last 22.67) - open/close 16222 (time 1787700963.73 total 879.65 last 23.86) - open/close 16505 (time 1787700973.73 total 889.65 last 28.29) - open/close 16704 (time 1787700983.74 total 899.67 last 19.87) - open/close 16919 (time 1787700993.75 total 909.67 last 21.49) - open/close 17147 (time 1787701003.98 total 919.91 last 22.28) - open/close 17320 (time 1787701015.79 total 931.72 last 14.65) - open/close 17556 (time 1787701026.27 total 942.20 last 22.51) - open/close 17800 (time 1787701036.96 total 952.89 last 22.82) - open/close 18312 (time 1787701046.98 total 962.91 last 51.11) - open/close 19035 (time 1787701057.01 total 972.93 last 72.12) - open/close 19748 (time 1787701067.02 total 982.94 last 71.23) - open/close 20000 (time 1787701070.20 total 986.13 last 79.16) - open/close 20743 (time 1787701080.21 total 996.14 last 74.21) - open/close 21469 (time 1787701090.22 total 1006.15 last 72.52) - open/close 22151 (time 1787701100.23 total 1016.15 last 68.17) - open/close 22789 (time 1787701110.24 total 1026.16 last 63.74) - open/close 23434 (time 1787701120.28 total 1036.20 last 64.24) - open/close 24137 (time 1787701130.29 total 1046.21 last 70.22) - open/close 24819 (time 1787701140.29 total 1056.22 last 68.17) - open/close 25476 (time 1787701150.31 total 1066.24 last 65.58) - open/close 26173 (time 1787701160.32 total 1076.24 last 69.66) - open/close 26821 (time 1787701170.32 total 1086.24 last 64.79) - open/close 27551 (time 1787701180.32 total 1096.25 last 72.97) - open/close 28211 (time 1787701190.33 total 1106.25 last 65.98) - open/close 28922 (time 1787701200.33 total 1116.26 last 71.07) - open/close 29631 (time 1787701210.35 total 1126.27 last 70.80) - open/close 30000 (time 1787701215.17 total 1131.09 last 76.53) - open/close 30726 (time 1787701225.17 total 1141.10 last 72.58) - open/close 31481 (time 1787701235.18 total 1151.10 last 75.46) - open/close 32213 (time 1787701245.18 total 1161.10 last 73.19) - open/close 32979 (time 1787701255.18 total 1171.11 last 76.56) - open/close 33673 (time 1787701265.19 total 1181.11 last 69.35) - open/close 34361 (time 1787701275.20 total 1191.12 last 68.72) - open/close 35034 (time 1787701285.22 total 1201.14 last 67.19) - open/close 35718 (time 1787701295.23 total 1211.15 last 68.32) - open/close 36419 (time 1787701305.23 total 1221.15 last 70.09) - open/close 37054 (time 1787701315.25 total 1231.17 last 63.37) - open/close 37750 (time 1787701325.26 total 1241.18 last 69.56) - open/close 38445 (time 1787701335.27 total 1251.20 last 69.38) - open/close 39127 (time 1787701345.27 total 1261.20 last 68.20) - open/close 39786 (time 1787701355.29 total 1271.21 last 65.80) - open/close 40000 (time 1787701358.19 total 1274.11 last 73.82) - open/close 40696 (time 1787701368.19 total 1284.11 last 69.59) - open/close 41327 (time 1787701378.20 total 1294.13 last 63.01) - open/close 42009 (time 1787701388.22 total 1304.14 last 68.12) - open/close 42770 (time 1787701398.22 total 1314.15 last 76.04) - open/close 43514 (time 1787701408.23 total 1324.15 last 74.37) - open/close 44242 (time 1787701418.23 total 1334.15 last 72.79) - open/close 44995 (time 1787701428.24 total 1344.16 last 75.21) - open/close 45703 (time 1787701438.25 total 1354.18 last 70.70) - open/close 46322 (time 1787701448.26 total 1364.18 last 61.87) - open/close 46975 (time 1787701458.27 total 1374.19 last 65.23) - open/close 47674 (time 1787701468.28 total 1384.20 last 69.83) - open/close 48327 (time 1787701478.29 total 1394.21 last 65.25) - open/close 48986 (time 1787701488.30 total 1404.22 last 65.83) - open/close 49655 (time 1787701498.30 total 1414.23 last 66.86) total: 50000 open/close in 1419.13 seconds: 35.23 ops/second - unlinked 0 (time 1787701507 ; total 0 ; last 0) - unlinked 10000 (time 1787701655 ; total 148 ; last 148) - unlinked 20000 (time 1787701803 ; total 296 ; last 148) - unlinked 30000 (time 1787701952 ; total 445 ; last 149) - unlinked 40000 (time 1787702095 ; total 588 ; last 143) total: 50000 unlinks in 740 seconds: 67.567566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg612-server unloading modules via unload_modules_local on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing unload_modules_local oleg612-server: modules unloaded. pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2