== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:56 (1788443996) start mds service on oleg619-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing load_modules_local oleg619-server: Loading modules from /home/green/git/lustre-release/lustre oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg619-server: detected 4 online CPUs by sysfs oleg619-server: MODOPTS_LIBCFS= oleg619-server: Force libcfs to create 2 CPU partitions oleg619-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg619-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg619-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg619-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 233 (time 1788444081.52 total 10.02 last 23.26) - open/close 400 (time 1788444091.53 total 20.03 last 16.68) - open/close 572 (time 1788444103.10 total 31.60 last 14.87) - open/close 638 (time 1788444114.31 total 42.81 last 5.89) - open/close 774 (time 1788444124.91 total 53.41 last 12.83) - open/close 952 (time 1788444135.17 total 63.67 last 17.36) - open/close 1185 (time 1788444145.35 total 73.85 last 22.88) - open/close 1419 (time 1788444155.84 total 84.34 last 22.30) - open/close 1640 (time 1788444166.33 total 94.83 last 21.07) - open/close 1901 (time 1788444176.47 total 104.97 last 25.75) - open/close 2111 (time 1788444186.47 total 114.97 last 20.99) - open/close 2274 (time 1788444196.51 total 125.01 last 16.24) - open/close 2388 (time 1788444209.01 total 137.51 last 9.12) - open/close 2611 (time 1788444219.03 total 147.53 last 22.25) - open/close 2752 (time 1788444229.38 total 157.88 last 13.63) - open/close 2991 (time 1788444240.38 total 168.88 last 21.73) - open/close 3150 (time 1788444252.29 total 180.79 last 13.35) - open/close 3302 (time 1788444262.32 total 190.82 last 15.16) - open/close 3503 (time 1788444273.22 total 201.72 last 18.45) - open/close 3550 (time 1788444283.83 total 212.33 last 4.43) - open/close 3737 (time 1788444293.85 total 222.35 last 18.67) - open/close 3969 (time 1788444304.38 total 232.88 last 22.03) - open/close 4188 (time 1788444314.83 total 243.33 last 20.95) - open/close 4419 (time 1788444324.84 total 253.34 last 23.07) - open/close 4606 (time 1788444334.97 total 263.47 last 18.47) - open/close 4720 (time 1788444345.86 total 274.36 last 10.47) - open/close 4911 (time 1788444359.93 total 288.43 last 13.57) - open/close 5059 (time 1788444370.94 total 299.44 last 13.45) - open/close 5305 (time 1788444381.90 total 310.40 last 22.44) - open/close 5553 (time 1788444391.92 total 320.42 last 24.76) - open/close 5772 (time 1788444402.24 total 330.74 last 21.22) - open/close 5968 (time 1788444412.88 total 341.38 last 18.42) - open/close 6077 (time 1788444423.67 total 352.17 last 10.10) - open/close 6304 (time 1788444434.41 total 362.91 last 21.14) - open/close 6523 (time 1788444445.37 total 373.87 last 19.99) - open/close 6732 (time 1788444455.49 total 383.99 last 20.65) - open/close 6998 (time 1788444465.52 total 394.02 last 26.51) - open/close 7135 (time 1788444477.38 total 405.88 last 11.55) - open/close 7361 (time 1788444487.39 total 415.90 last 22.56) - open/close 7568 (time 1788444497.71 total 426.21 last 20.06) - open/close 7815 (time 1788444510.66 total 439.16 last 19.08) - open/close 7923 (time 1788444521.40 total 449.90 last 10.05) - open/close 8082 (time 1788444531.63 total 460.13 last 15.54) - open/close 8298 (time 1788444541.77 total 470.27 last 21.31) - open/close 8558 (time 1788444552.12 total 480.62 last 25.11) - open/close 8809 (time 1788444562.48 total 490.99 last 24.22) - open/close 9065 (time 1788444572.49 total 500.99 last 25.59) - open/close 9178 (time 1788444582.49 total 510.99 last 11.30) - open/close 9241 (time 1788444592.50 total 521.00 last 6.30) - open/close 9433 (time 1788444602.79 total 531.29 last 18.66) - open/close 9596 (time 1788444613.38 total 541.88 last 15.38) - open/close 9781 (time 1788444623.67 total 552.17 last 17.98) - open/close 10000 (time 1788444632.53 total 561.03 last 24.73) - open/close 10222 (time 1788444643.13 total 571.63 last 20.93) - open/close 10462 (time 1788444653.14 total 581.64 last 23.98) - open/close 10720 (time 1788444663.14 total 591.64 last 25.80) - open/close 10953 (time 1788444673.15 total 601.65 last 23.29) - open/close 11174 (time 1788444683.44 total 611.94 last 21.47) - open/close 11406 (time 1788444694.01 total 622.51 last 21.96) - open/close 11636 (time 1788444704.37 total 632.87 last 22.19) - open/close 11897 (time 1788444714.39 total 642.89 last 26.06) - open/close 12076 (time 1788444725.13 total 653.63 last 16.67) - open/close 12323 (time 1788444735.45 total 663.95 last 23.94) - open/close 12513 (time 1788444745.82 total 674.32 last 18.32) - open/close 12753 (time 1788444756.41 total 684.91 last 22.66) - open/close 13009 (time 1788444766.92 total 695.42 last 24.35) - open/close 13215 (time 1788444777.72 total 706.22 last 19.08) - open/close 13483 (time 1788444787.73 total 716.23 last 26.77) - open/close 13737 (time 1788444797.73 total 726.23 last 25.39) - open/close 13865 (time 1788444808.15 total 736.65 last 12.29) - open/close 14020 (time 1788444818.37 total 746.87 last 15.17) - open/close 14167 (time 1788444828.37 total 756.87 last 14.69) - open/close 14379 (time 1788444838.37 total 766.88 last 21.19) - open/close 14558 (time 1788444849.05 total 777.55 last 16.77) - open/close 14805 (time 1788444859.45 total 787.95 last 23.75) - open/close 15077 (time 1788444869.45 total 797.95 last 27.18) - open/close 15289 (time 1788444879.67 total 808.17 last 20.75) - open/close 15459 (time 1788444890.55 total 819.05 last 15.63) - open/close 15651 (time 1788444900.57 total 829.07 last 19.17) - open/close 15773 (time 1788444912.81 total 841.31 last 9.96) - open/close 15893 (time 1788444929.82 total 858.32 last 7.06) - open/close 16177 (time 1788444939.85 total 868.35 last 28.33) - open/close 16396 (time 1788444950.83 total 879.33 last 19.93) - open/close 16646 (time 1788444960.85 total 889.35 last 24.96) - open/close 16897 (time 1788444971.23 total 899.73 last 24.19) - open/close 17105 (time 1788444981.82 total 910.32 last 19.63) - open/close 17312 (time 1788444992.44 total 920.94 last 19.50) - open/close 17610 (time 1788445005.32 total 933.82 last 23.13) - open/close 17731 (time 1788445015.64 total 944.14 last 11.73) - open/close 17827 (time 1788445026.20 total 954.70 last 9.09) - open/close 18188 (time 1788445036.21 total 964.71 last 36.08) - open/close 18846 (time 1788445046.23 total 974.73 last 65.66) - open/close 19601 (time 1788445056.25 total 984.75 last 75.31) - open/close 20000 (time 1788445061.88 total 990.38 last 70.89) - open/close 20708 (time 1788445071.88 total 1000.38 last 70.77) - open/close 21368 (time 1788445081.89 total 1010.39 last 65.99) - open/close 22035 (time 1788445091.89 total 1020.39 last 66.67) - open/close 22660 (time 1788445101.90 total 1030.40 last 62.47) - open/close 23278 (time 1788445111.92 total 1040.42 last 61.64) - open/close 23940 (time 1788445121.94 total 1050.44 last 66.10) - open/close 24591 (time 1788445131.95 total 1060.45 last 65.04) - open/close 25285 (time 1788445141.95 total 1070.45 last 69.35) - open/close 25939 (time 1788445151.96 total 1080.46 last 65.34) - open/close 26618 (time 1788445161.97 total 1090.47 last 67.84) - open/close 27291 (time 1788445171.99 total 1100.49 last 67.17) - open/close 27964 (time 1788445182.00 total 1110.50 last 67.22) - open/close 28655 (time 1788445192.01 total 1120.51 last 69.07) - open/close 29356 (time 1788445202.01 total 1130.51 last 70.10) - open/close 30000 (time 1788445211.83 total 1140.33 last 65.53) - open/close 30581 (time 1788445221.88 total 1150.38 last 57.86) - open/close 31235 (time 1788445231.90 total 1160.40 last 65.25) - open/close 31724 (time 1788445241.90 total 1170.40 last 48.88) - open/close 32176 (time 1788445251.92 total 1180.42 last 45.11) - open/close 32741 (time 1788445261.92 total 1190.42 last 56.49) - open/close 33418 (time 1788445271.94 total 1200.44 last 67.62) - open/close 34117 (time 1788445281.94 total 1210.44 last 69.87) - open/close 34829 (time 1788445291.96 total 1220.46 last 71.05) - open/close 35563 (time 1788445301.97 total 1230.47 last 73.32) - open/close 36292 (time 1788445311.98 total 1240.48 last 72.85) - open/close 37047 (time 1788445321.99 total 1250.49 last 75.41) - open/close 37817 (time 1788445332.00 total 1260.50 last 76.96) - open/close 38633 (time 1788445342.02 total 1270.52 last 81.42) - open/close 39363 (time 1788445352.03 total 1280.53 last 72.95) - open/close 40000 (time 1788445360.86 total 1289.36 last 72.12) - open/close 40620 (time 1788445370.86 total 1299.36 last 62.00) - open/close 41365 (time 1788445380.86 total 1309.36 last 74.50) - open/close 42072 (time 1788445390.87 total 1319.37 last 70.61) - open/close 42697 (time 1788445400.89 total 1329.39 last 62.42) - open/close 43356 (time 1788445410.89 total 1339.39 last 65.87) - open/close 43928 (time 1788445420.89 total 1349.39 last 57.19) - open/close 44460 (time 1788445430.90 total 1359.40 last 53.18) - open/close 45023 (time 1788445440.95 total 1369.45 last 56.02) - open/close 45626 (time 1788445450.95 total 1379.45 last 60.26) - open/close 46242 (time 1788445460.97 total 1389.47 last 61.50) - open/close 46847 (time 1788445470.98 total 1399.48 last 60.45) - open/close 47507 (time 1788445480.98 total 1409.48 last 65.96) - open/close 48085 (time 1788445490.99 total 1419.49 last 57.74) - open/close 48708 (time 1788445501.01 total 1429.51 last 62.21) - open/close 49349 (time 1788445511.02 total 1439.52 last 64.02) - open/close 49982 (time 1788445521.02 total 1449.52 last 63.28) total: 50000 open/close in 1449.83 seconds: 34.49 ops/second - unlinked 0 (time 1788445525 ; total 0 ; last 0) - unlinked 10000 (time 1788445691 ; total 166 ; last 166) - unlinked 20000 (time 1788445850 ; total 325 ; last 159) - unlinked 30000 (time 1788446013 ; total 488 ; last 163) - unlinked 40000 (time 1788446161 ; total 636 ; last 148) total: 50000 unlinks in 785 seconds: 63.694267 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg619-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg619-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server unloading modules via unload_modules_local on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing unload_modules_local oleg619-server: modules unloaded. pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2