== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:04:16 (1786712656) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 126 (time 1786712744.90 total 10.00 last 12.60) - open/close 360 (time 1786712755.41 total 20.51 last 22.26) - open/close 506 (time 1786712765.46 total 30.57 last 14.53) - open/close 614 (time 1786712776.16 total 41.26 last 10.10) - open/close 763 (time 1786712787.68 total 52.78 last 12.93) - open/close 895 (time 1786712797.91 total 63.01 last 12.90) - open/close 1031 (time 1786712807.92 total 73.02 last 13.59) - open/close 1154 (time 1786712817.92 total 83.02 last 12.30) - open/close 1170 (time 1786712828.63 total 93.73 last 1.49) - open/close 1377 (time 1786712838.63 total 103.73 last 20.69) - open/close 1559 (time 1786712848.66 total 113.76 last 18.15) - open/close 1784 (time 1786712859.86 total 124.96 last 20.09) - open/close 1884 (time 1786712869.88 total 134.98 last 9.98) - open/close 2026 (time 1786712879.89 total 144.99 last 14.19) - open/close 2131 (time 1786712889.94 total 155.04 last 10.44) - open/close 2228 (time 1786712899.95 total 165.06 last 9.69) - open/close 2375 (time 1786712912.17 total 177.27 last 12.03) - open/close 2603 (time 1786712922.18 total 187.28 last 22.79) - open/close 2833 (time 1786712932.20 total 197.30 last 22.95) - open/close 3042 (time 1786712942.42 total 207.53 last 20.44) - open/close 3272 (time 1786712952.43 total 217.53 last 22.99) - open/close 3443 (time 1786712963.29 total 228.39 last 15.75) - open/close 3630 (time 1786712973.63 total 238.73 last 18.09) - open/close 3805 (time 1786712983.76 total 248.86 last 17.28) - open/close 4038 (time 1786712993.76 total 258.86 last 23.29) - open/close 4275 (time 1786713004.78 total 269.88 last 21.50) - open/close 4424 (time 1786713015.33 total 280.43 last 14.13) - open/close 4600 (time 1786713025.37 total 290.47 last 17.53) - open/close 4634 (time 1786713035.37 total 300.48 last 3.40) - open/close 4927 (time 1786713045.73 total 310.83 last 28.30) - open/close 5064 (time 1786713055.83 total 320.93 last 13.56) - open/close 5167 (time 1786713066.14 total 331.24 last 9.99) - open/close 5308 (time 1786713076.79 total 341.89 last 13.24) - open/close 5407 (time 1786713087.29 total 352.39 last 9.43) - open/close 5573 (time 1786713098.12 total 363.22 last 15.33) - open/close 5769 (time 1786713108.87 total 373.97 last 18.24) - open/close 5950 (time 1786713118.88 total 383.98 last 18.08) - open/close 6082 (time 1786713129.57 total 394.67 last 12.35) - open/close 6142 (time 1786713139.57 total 404.67 last 6.00) - open/close 6267 (time 1786713151.33 total 416.43 last 10.63) - open/close 6499 (time 1786713162.28 total 427.38 last 21.20) - open/close 6733 (time 1786713173.03 total 438.13 last 21.76) - open/close 6973 (time 1786713183.03 total 448.14 last 23.99) - open/close 7215 (time 1786713193.76 total 458.86 last 22.57) - open/close 7370 (time 1786713203.78 total 468.88 last 15.46) - open/close 7535 (time 1786713214.37 total 479.47 last 15.58) - open/close 7834 (time 1786713224.41 total 489.51 last 29.77) - open/close 8015 (time 1786713235.44 total 500.54 last 16.41) - open/close 8236 (time 1786713245.54 total 510.64 last 21.88) - open/close 8397 (time 1786713255.55 total 520.65 last 16.09) - open/close 8549 (time 1786713266.07 total 531.18 last 14.44) - open/close 8839 (time 1786713276.97 total 542.07 last 26.62) - open/close 9071 (time 1786713289.26 total 554.36 last 18.88) - open/close 9278 (time 1786713299.27 total 564.37 last 20.67) - open/close 9514 (time 1786713309.27 total 574.38 last 23.59) - open/close 9712 (time 1786713319.70 total 584.80 last 18.99) - open/close 9959 (time 1786713329.82 total 594.92 last 24.41) - open/close 10216 (time 1786713342.56 total 607.67 last 20.16) - open/close 10368 (time 1786713353.33 total 618.43 last 14.12) - open/close 10536 (time 1786713363.35 total 628.45 last 16.76) - open/close 10761 (time 1786713374.07 total 639.18 last 20.98) - open/close 10900 (time 1786713384.66 total 649.76 last 13.13) - open/close 11055 (time 1786713395.56 total 660.66 last 14.23) - open/close 11274 (time 1786713405.56 total 670.66 last 21.89) - open/close 11401 (time 1786713416.61 total 681.71 last 11.49) - open/close 11593 (time 1786713427.18 total 692.28 last 18.17) - open/close 11746 (time 1786713437.94 total 703.04 last 14.22) - open/close 12013 (time 1786713447.95 total 713.05 last 26.67) - open/close 12208 (time 1786713457.95 total 723.05 last 19.49) - open/close 12388 (time 1786713467.97 total 733.07 last 17.96) - open/close 12545 (time 1786713478.61 total 743.71 last 14.76) - open/close 12750 (time 1786713489.51 total 754.62 last 18.80) - open/close 12925 (time 1786713500.40 total 765.50 last 16.07) - open/close 13162 (time 1786713511.26 total 776.36 last 21.83) - open/close 13345 (time 1786713522.27 total 787.37 last 16.62) - open/close 13445 (time 1786713532.91 total 798.01 last 9.39) - open/close 13679 (time 1786713542.94 total 808.04 last 23.35) - open/close 13846 (time 1786713552.94 total 818.04 last 16.70) - open/close 14014 (time 1786713563.10 total 828.20 last 16.53) - open/close 14270 (time 1786713574.01 total 839.11 last 23.45) - open/close 14495 (time 1786713584.03 total 849.13 last 22.46) - open/close 14724 (time 1786713594.24 total 859.34 last 22.43) - open/close 14891 (time 1786713604.97 total 870.07 last 15.56) - open/close 15007 (time 1786713614.98 total 880.08 last 11.60) - open/close 15149 (time 1786713625.53 total 890.63 last 13.46) - open/close 15349 (time 1786713636.13 total 901.23 last 18.87) - open/close 15599 (time 1786713646.15 total 911.25 last 24.95) - open/close 15749 (time 1786713656.15 total 921.25 last 14.99) - open/close 15919 (time 1786713667.03 total 932.13 last 15.63) - open/close 15985 (time 1786713678.94 total 944.04 last 5.54) - open/close 16111 (time 1786713688.94 total 954.04 last 12.60) - open/close 16324 (time 1786713698.94 total 964.04 last 21.30) - open/close 16493 (time 1786713708.95 total 974.05 last 16.88) - open/close 16689 (time 1786713719.27 total 984.37 last 18.98) - open/close 16900 (time 1786713729.89 total 994.99 last 19.87) - open/close 17117 (time 1786713740.83 total 1005.93 last 19.85) - open/close 17263 (time 1786713750.85 total 1015.95 last 14.57) - open/close 17599 (time 1786713760.86 total 1025.96 last 33.55) - open/close 18210 (time 1786713770.87 total 1035.98 last 61.03) - open/close 18856 (time 1786713780.88 total 1045.98 last 64.57) - open/close 19553 (time 1786713790.89 total 1055.99 last 69.65) - open/close 20000 (time 1786713796.97 total 1062.07 last 73.44) - open/close 20674 (time 1786713806.98 total 1072.08 last 67.38) - open/close 21282 (time 1786713816.99 total 1082.09 last 60.75) - open/close 21921 (time 1786713827.00 total 1092.10 last 63.82) - open/close 22532 (time 1786713837.02 total 1102.12 last 60.99) - open/close 23107 (time 1786713847.02 total 1112.12 last 57.49) - open/close 23684 (time 1786713857.03 total 1122.13 last 57.62) - open/close 24170 (time 1786713867.05 total 1132.15 last 48.53) - open/close 24777 (time 1786713877.05 total 1142.15 last 60.68) - open/close 25508 (time 1786713887.05 total 1152.15 last 73.09) - open/close 26103 (time 1786713897.06 total 1162.16 last 59.47) - open/close 26767 (time 1786713907.06 total 1172.16 last 66.39) - open/close 27524 (time 1786713917.06 total 1182.16 last 75.68) - open/close 28203 (time 1786713927.07 total 1192.17 last 67.82) - open/close 28900 (time 1786713937.08 total 1202.18 last 69.65) - open/close 29634 (time 1786713947.09 total 1212.19 last 73.34) - open/close 30000 (time 1786713952.07 total 1217.18 last 73.40) - open/close 30695 (time 1786713962.07 total 1227.18 last 69.50) - open/close 31422 (time 1786713972.08 total 1237.18 last 72.67) - open/close 32104 (time 1786713982.08 total 1247.18 last 68.17) - open/close 32818 (time 1786713992.09 total 1257.19 last 71.34) - open/close 33536 (time 1786714002.10 total 1267.20 last 71.73) - open/close 34111 (time 1786714012.12 total 1277.22 last 57.41) - open/close 34694 (time 1786714022.12 total 1287.22 last 58.29) - open/close 35369 (time 1786714032.12 total 1297.22 last 67.49) - open/close 35988 (time 1786714042.12 total 1307.22 last 61.88) - open/close 36572 (time 1786714052.13 total 1317.23 last 58.38) - open/close 37233 (time 1786714062.14 total 1327.24 last 66.02) - open/close 37754 (time 1786714072.15 total 1337.25 last 52.06) - open/close 38392 (time 1786714082.15 total 1347.25 last 63.76) - open/close 39065 (time 1786714092.16 total 1357.26 last 67.27) - open/close 39765 (time 1786714102.16 total 1367.26 last 69.97) - open/close 40000 (time 1786714106.13 total 1371.23 last 59.19) - open/close 40735 (time 1786714116.14 total 1381.24 last 73.47) - open/close 41371 (time 1786714126.16 total 1391.26 last 63.46) - open/close 42028 (time 1786714136.17 total 1401.27 last 65.63) - open/close 42676 (time 1786714146.17 total 1411.27 last 64.79) - open/close 43360 (time 1786714156.18 total 1421.29 last 68.31) - open/close 43971 (time 1786714166.20 total 1431.30 last 61.02) - open/close 44544 (time 1786714176.22 total 1441.32 last 57.19) - open/close 45122 (time 1786714186.23 total 1451.33 last 57.75) - open/close 45687 (time 1786714196.24 total 1461.34 last 56.45) - open/close 46296 (time 1786714206.24 total 1471.34 last 60.88) - open/close 46999 (time 1786714216.25 total 1481.35 last 70.22) - open/close 47586 (time 1786714226.25 total 1491.35 last 58.70) - open/close 48264 (time 1786714236.25 total 1501.36 last 67.77) - open/close 48930 (time 1786714246.26 total 1511.36 last 66.54) - open/close 49547 (time 1786714256.27 total 1521.37 last 61.64) total: 50000 open/close in 1529.13 seconds: 32.70 ops/second - unlinked 0 (time 1786714269 ; total 0 ; last 0) - unlinked 10000 (time 1786714431 ; total 162 ; last 162) - unlinked 20000 (time 1786714591 ; total 322 ; last 160) - unlinked 30000 (time 1786714763 ; total 494 ; last 172) - unlinked 40000 (time 1786714937 ; total 668 ; last 174) total: 50000 unlinks in 842 seconds: 59.382423 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2