== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:29:32 (1789482572) start mds service on oleg309-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing load_modules_local oleg309-server: Loading modules from /home/green/git/lustre-release/lustre oleg309-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg309-server: detected 4 online CPUs by sysfs oleg309-server: MODOPTS_LIBCFS= oleg309-server: Force libcfs to create 2 CPU partitions oleg309-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg309-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg309-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg309-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg309-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 178 (time 1789482675.61 total 10.01 last 17.79) - open/close 306 (time 1789482686.00 total 20.40 last 12.32) - open/close 433 (time 1789482697.32 total 31.72 last 11.22) - open/close 619 (time 1789482708.46 total 42.86 last 16.69) - open/close 796 (time 1789482719.07 total 53.47 last 16.68) - open/close 980 (time 1789482729.11 total 63.50 last 18.34) - open/close 1234 (time 1789482739.80 total 74.20 last 23.75) - open/close 1477 (time 1789482749.91 total 84.31 last 24.05) - open/close 1701 (time 1789482759.94 total 94.33 last 22.34) - open/close 1919 (time 1789482770.16 total 104.56 last 21.32) - open/close 2210 (time 1789482781.16 total 115.56 last 26.46) - open/close 2538 (time 1789482791.18 total 125.57 last 32.75) - open/close 2794 (time 1789482801.55 total 135.95 last 24.67) - open/close 2887 (time 1789482812.40 total 146.80 last 8.57) - open/close 3206 (time 1789482822.40 total 156.80 last 31.89) - open/close 3426 (time 1789482832.93 total 167.33 last 20.90) - open/close 3632 (time 1789482843.18 total 177.58 last 20.10) - open/close 3776 (time 1789482853.50 total 187.89 last 13.96) - open/close 3904 (time 1789482863.64 total 198.04 last 12.62) - open/close 4038 (time 1789482873.65 total 208.05 last 13.39) - open/close 4194 (time 1789482883.65 total 218.05 last 15.60) - open/close 4329 (time 1789482893.65 total 228.05 last 13.50) - open/close 4383 (time 1789482903.66 total 238.06 last 5.39) - open/close 4480 (time 1789482913.88 total 248.28 last 9.49) - open/close 4618 (time 1789482923.89 total 258.29 last 13.79) - open/close 4754 (time 1789482933.89 total 268.29 last 13.60) - open/close 4925 (time 1789482943.91 total 278.30 last 17.07) - open/close 5222 (time 1789482954.98 total 289.38 last 26.82) - open/close 5475 (time 1789482965.47 total 299.87 last 24.11) - open/close 5749 (time 1789482977.94 total 312.34 last 21.97) - open/close 5966 (time 1789482987.95 total 322.35 last 21.68) - open/close 6211 (time 1789482997.96 total 332.36 last 24.47) - open/close 6465 (time 1789483007.97 total 342.37 last 25.37) - open/close 6710 (time 1789483017.97 total 352.37 last 24.50) - open/close 7046 (time 1789483027.98 total 362.38 last 33.57) - open/close 7329 (time 1789483037.99 total 372.39 last 28.28) - open/close 7585 (time 1789483048.48 total 382.88 last 24.39) - open/close 7809 (time 1789483058.52 total 392.92 last 22.32) - open/close 8034 (time 1789483068.53 total 402.92 last 22.48) - open/close 8262 (time 1789483078.53 total 412.93 last 22.79) - open/close 8522 (time 1789483088.54 total 422.94 last 25.98) - open/close 8767 (time 1789483099.05 total 433.44 last 23.32) - open/close 9037 (time 1789483109.52 total 443.92 last 25.78) - open/close 9282 (time 1789483119.53 total 453.93 last 24.47) - open/close 9556 (time 1789483129.56 total 463.96 last 27.31) - open/close 9890 (time 1789483139.58 total 473.98 last 33.35) - open/close 10000 (time 1789483144.29 total 478.69 last 23.35) - open/close 10187 (time 1789483154.30 total 488.69 last 18.69) - open/close 10401 (time 1789483164.30 total 498.70 last 21.39) - open/close 10652 (time 1789483175.78 total 510.17 last 21.87) - open/close 10899 (time 1789483186.09 total 520.48 last 23.96) - open/close 11108 (time 1789483196.10 total 530.50 last 20.87) - open/close 11265 (time 1789483206.10 total 540.50 last 15.70) - open/close 11337 (time 1789483216.28 total 550.68 last 7.07) - open/close 11445 (time 1789483226.46 total 560.86 last 10.61) - open/close 11641 (time 1789483236.47 total 570.87 last 19.58) - open/close 11766 (time 1789483246.48 total 580.87 last 12.49) - open/close 11969 (time 1789483257.31 total 591.71 last 18.74) - open/close 12078 (time 1789483267.54 total 601.94 last 10.66) - open/close 12375 (time 1789483277.92 total 612.32 last 28.61) - open/close 12612 (time 1789483288.41 total 622.81 last 22.59) - open/close 12882 (time 1789483298.45 total 632.85 last 26.88) - open/close 13185 (time 1789483308.47 total 642.87 last 30.24) - open/close 13373 (time 1789483318.47 total 652.87 last 18.80) - open/close 13633 (time 1789483328.67 total 663.07 last 25.49) - open/close 13932 (time 1789483338.68 total 673.08 last 29.86) - open/close 14193 (time 1789483349.47 total 683.87 last 24.20) - open/close 14409 (time 1789483359.76 total 694.16 last 20.99) - open/close 14660 (time 1789483369.87 total 704.27 last 24.82) - open/close 14905 (time 1789483380.53 total 714.93 last 22.99) - open/close 15114 (time 1789483393.37 total 727.77 last 16.27) - open/close 15354 (time 1789483403.37 total 737.77 last 24.00) - open/close 15604 (time 1789483413.38 total 747.78 last 24.97) - open/close 15903 (time 1789483424.18 total 758.58 last 27.69) - open/close 16179 (time 1789483434.20 total 768.60 last 27.54) - open/close 16428 (time 1789483444.40 total 778.80 last 24.41) - open/close 16648 (time 1789483454.41 total 788.81 last 21.97) - open/close 16750 (time 1789483464.42 total 798.81 last 10.20) - open/close 16953 (time 1789483474.43 total 808.83 last 20.27) - open/close 17207 (time 1789483485.17 total 819.57 last 23.65) - open/close 17542 (time 1789483495.52 total 829.92 last 32.35) - open/close 17800 (time 1789483505.56 total 839.96 last 25.71) - open/close 18078 (time 1789483515.94 total 850.34 last 26.78) - open/close 18319 (time 1789483526.13 total 860.52 last 23.65) - open/close 18490 (time 1789483536.48 total 870.88 last 16.51) - open/close 18757 (time 1789483546.78 total 881.17 last 25.94) - open/close 19074 (time 1789483556.79 total 891.19 last 31.66) - open/close 19301 (time 1789483567.82 total 902.22 last 20.58) - open/close 19613 (time 1789483577.83 total 912.23 last 31.16) - open/close 19894 (time 1789483588.38 total 922.77 last 26.65) - open/close 20000 (time 1789483591.17 total 925.57 last 37.90) - open/close 20256 (time 1789483603.69 total 938.08 last 20.46) - open/close 21120 (time 1789483613.70 total 948.09 last 86.31) - open/close 21965 (time 1789483623.72 total 958.12 last 84.31) - open/close 22748 (time 1789483633.72 total 968.12 last 78.29) - open/close 23557 (time 1789483643.72 total 978.12 last 80.86) - open/close 24208 (time 1789483653.74 total 988.13 last 65.03) - open/close 24940 (time 1789483663.74 total 998.14 last 73.18) - open/close 25812 (time 1789483673.76 total 1008.16 last 87.03) - open/close 26770 (time 1789483683.77 total 1018.17 last 95.65) - open/close 27620 (time 1789483693.78 total 1028.17 last 84.99) - open/close 28332 (time 1789483703.79 total 1038.19 last 71.10) - open/close 29160 (time 1789483713.80 total 1048.20 last 82.69) - open/close 30000 (time 1789483723.07 total 1057.47 last 90.61) - open/close 30935 (time 1789483733.08 total 1067.48 last 93.44) - open/close 31854 (time 1789483743.09 total 1077.49 last 91.81) - open/close 32833 (time 1789483753.09 total 1087.49 last 97.88) - open/close 33665 (time 1789483763.09 total 1097.49 last 83.20) - open/close 34577 (time 1789483773.10 total 1107.49 last 91.16) - open/close 35566 (time 1789483783.10 total 1117.50 last 98.85) - open/close 36479 (time 1789483793.10 total 1127.50 last 91.27) - open/close 37437 (time 1789483803.11 total 1137.51 last 95.73) - open/close 38358 (time 1789483813.12 total 1147.52 last 92.03) - open/close 39303 (time 1789483823.12 total 1157.52 last 94.49) - open/close 40000 (time 1789483829.88 total 1164.28 last 103.03) - open/close 40925 (time 1789483839.89 total 1174.29 last 92.40) - open/close 41899 (time 1789483849.90 total 1184.30 last 97.36) - open/close 42810 (time 1789483859.91 total 1194.31 last 91.01) - open/close 43713 (time 1789483869.92 total 1204.32 last 90.22) - open/close 44688 (time 1789483879.92 total 1214.32 last 97.49) - open/close 45579 (time 1789483889.93 total 1224.32 last 89.04) - open/close 46482 (time 1789483899.93 total 1234.33 last 90.23) - open/close 47457 (time 1789483909.93 total 1244.33 last 97.48) - open/close 48352 (time 1789483919.94 total 1254.34 last 89.44) - open/close 49301 (time 1789483929.95 total 1264.34 last 94.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1267.33 seconds: 39.16 ops/second - unlinked 0 (time 1789483936 ; total 0 ; last 0) - unlinked 10000 (time 1789484052 ; total 116 ; last 116) - unlinked 20000 (time 1789484158 ; total 222 ; last 106) - unlinked 30000 (time 1789484218 ; total 282 ; last 60) - unlinked 40000 (time 1789484283 ; total 347 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 405 seconds: 122.545677 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg309-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg309-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg309-server unloading modules via unload_modules_local on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing unload_modules_local oleg309-server: modules unloaded. pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2