== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:59:45 (1788310785) start mds service on oleg254-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg254-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 191 (time 1788310915.44 total 11.62 last 16.43) - open/close 315 (time 1788310926.45 total 22.63 last 11.27) - open/close 598 (time 1788310936.76 total 32.94 last 27.44) - open/close 746 (time 1788310946.77 total 42.95 last 14.78) - open/close 914 (time 1788310957.36 total 53.54 last 15.88) - open/close 1027 (time 1788310967.38 total 63.56 last 11.28) - open/close 1118 (time 1788310977.39 total 73.57 last 9.08) - open/close 1344 (time 1788310988.27 total 84.45 last 20.79) - open/close 1583 (time 1788310998.27 total 94.45 last 23.88) - open/close 1663 (time 1788311008.28 total 104.46 last 7.99) - open/close 1883 (time 1788311018.30 total 114.48 last 21.97) - open/close 2167 (time 1788311028.31 total 124.49 last 28.38) - open/close 2394 (time 1788311038.31 total 134.49 last 22.70) - open/close 2568 (time 1788311048.32 total 144.50 last 17.37) - open/close 2731 (time 1788311058.63 total 154.81 last 15.81) - open/close 2991 (time 1788311068.64 total 164.82 last 25.98) - open/close 3245 (time 1788311078.65 total 174.83 last 25.37) - open/close 3399 (time 1788311088.75 total 184.93 last 15.24) - open/close 3604 (time 1788311098.76 total 194.94 last 20.50) - open/close 3829 (time 1788311108.77 total 204.95 last 22.48) - open/close 3918 (time 1788311119.15 total 215.33 last 8.57) - open/close 4053 (time 1788311129.71 total 225.89 last 12.79) - open/close 4236 (time 1788311139.75 total 235.93 last 18.23) - open/close 4490 (time 1788311150.18 total 246.35 last 24.36) - open/close 4667 (time 1788311160.82 total 257.00 last 16.63) - open/close 4803 (time 1788311171.49 total 267.67 last 12.74) - open/close 4920 (time 1788311182.26 total 278.44 last 10.87) - open/close 5067 (time 1788311194.42 total 290.60 last 12.09) - open/close 5273 (time 1788311205.12 total 301.30 last 19.25) - open/close 5488 (time 1788311215.61 total 311.79 last 20.49) - open/close 5638 (time 1788311225.96 total 322.14 last 14.49) - open/close 5946 (time 1788311235.98 total 332.16 last 30.75) - open/close 6028 (time 1788311246.21 total 342.39 last 8.01) - open/close 6242 (time 1788311256.50 total 352.68 last 20.81) - open/close 6469 (time 1788311267.05 total 363.23 last 21.51) - open/close 6667 (time 1788311277.07 total 373.25 last 19.75) - open/close 6852 (time 1788311287.22 total 383.40 last 18.22) - open/close 6993 (time 1788311297.45 total 393.63 last 13.79) - open/close 7237 (time 1788311307.94 total 404.12 last 23.25) - open/close 7473 (time 1788311317.96 total 414.14 last 23.54) - open/close 7673 (time 1788311328.60 total 424.78 last 18.80) - open/close 7845 (time 1788311341.68 total 437.86 last 13.15) - open/close 7932 (time 1788311351.69 total 447.87 last 8.70) - open/close 8131 (time 1788311362.74 total 458.92 last 18.01) - open/close 8187 (time 1788311372.85 total 469.03 last 5.54) - open/close 8363 (time 1788311383.92 total 480.10 last 15.90) - open/close 8533 (time 1788311393.99 total 490.17 last 16.88) - open/close 8655 (time 1788311405.05 total 501.23 last 11.03) - open/close 8871 (time 1788311415.59 total 511.77 last 20.49) - open/close 9123 (time 1788311426.62 total 522.80 last 22.84) - open/close 9249 (time 1788311436.65 total 532.83 last 12.57) - open/close 9369 (time 1788311446.67 total 542.85 last 11.97) - open/close 9643 (time 1788311457.14 total 553.32 last 26.17) - open/close 9906 (time 1788311467.17 total 563.35 last 26.23) - open/close 10000 (time 1788311470.89 total 567.07 last 25.27) - open/close 10237 (time 1788311483.25 total 579.43 last 19.17) - open/close 10539 (time 1788311494.06 total 590.24 last 27.94) - open/close 10772 (time 1788311504.10 total 600.28 last 23.21) - open/close 10884 (time 1788311514.73 total 610.91 last 10.54) - open/close 11142 (time 1788311524.74 total 620.91 last 25.79) - open/close 11329 (time 1788311536.04 total 632.22 last 16.54) - open/close 11497 (time 1788311546.90 total 643.08 last 15.47) - open/close 11614 (time 1788311557.74 total 653.92 last 10.80) - open/close 11756 (time 1788311567.98 total 664.16 last 13.86) - open/close 11978 (time 1788311577.99 total 674.17 last 22.18) - open/close 12236 (time 1788311588.94 total 685.12 last 23.58) - open/close 12510 (time 1788311598.94 total 695.12 last 27.40) - open/close 12796 (time 1788311608.94 total 705.12 last 28.58) - open/close 13017 (time 1788311618.95 total 715.13 last 22.09) - open/close 13288 (time 1788311629.62 total 725.80 last 25.38) - open/close 13554 (time 1788311639.86 total 736.04 last 25.98) - open/close 13811 (time 1788311650.79 total 746.97 last 23.53) - open/close 13962 (time 1788311660.80 total 756.98 last 15.08) - open/close 14182 (time 1788311671.09 total 767.27 last 21.38) - open/close 14301 (time 1788311681.83 total 778.01 last 11.08) - open/close 14472 (time 1788311692.17 total 788.35 last 16.55) - open/close 14650 (time 1788311702.27 total 798.45 last 17.62) - open/close 14846 (time 1788311712.44 total 808.62 last 19.28) - open/close 15178 (time 1788311722.44 total 818.62 last 33.19) - open/close 15431 (time 1788311733.50 total 829.68 last 22.87) - open/close 15629 (time 1788311743.99 total 840.17 last 18.88) - open/close 15880 (time 1788311754.95 total 851.13 last 22.89) - open/close 16111 (time 1788311764.96 total 861.14 last 23.08) - open/close 16292 (time 1788311776.14 total 872.32 last 16.19) - open/close 16517 (time 1788311786.15 total 882.33 last 22.49) - open/close 16691 (time 1788311796.16 total 892.34 last 17.39) - open/close 16788 (time 1788311808.59 total 904.77 last 7.80) - open/close 17035 (time 1788311819.30 total 915.48 last 23.07) - open/close 17305 (time 1788311829.31 total 925.49 last 26.98) - open/close 17562 (time 1788311839.45 total 935.63 last 25.35) - open/close 17830 (time 1788311849.52 total 945.70 last 26.59) - open/close 18022 (time 1788311859.54 total 955.72 last 19.17) - open/close 18289 (time 1788311870.12 total 966.30 last 25.23) - open/close 18456 (time 1788311880.44 total 976.62 last 16.19) - open/close 18598 (time 1788311890.75 total 986.93 last 13.77) - open/close 18693 (time 1788311903.09 total 999.27 last 7.70) - open/close 18827 (time 1788311915.70 total 1011.88 last 10.63) - open/close 18988 (time 1788311926.08 total 1022.26 last 15.51) - open/close 19573 (time 1788311936.08 total 1032.26 last 58.47) - open/close 20000 (time 1788311940.02 total 1036.20 last 108.55) - open/close 20848 (time 1788311950.03 total 1046.21 last 84.68) - open/close 21703 (time 1788311960.04 total 1056.22 last 85.46) - open/close 22391 (time 1788311970.04 total 1066.22 last 68.80) - open/close 23245 (time 1788311980.04 total 1076.22 last 85.35) - open/close 24174 (time 1788311990.05 total 1086.23 last 92.88) - open/close 25095 (time 1788312000.06 total 1096.24 last 91.97) - open/close 26041 (time 1788312010.06 total 1106.24 last 94.58) - open/close 26974 (time 1788312020.07 total 1116.25 last 93.24) - open/close 27800 (time 1788312030.08 total 1126.26 last 82.51) - open/close 28799 (time 1788312040.08 total 1136.26 last 99.83) - open/close 29866 (time 1788312050.09 total 1146.27 last 106.62) - open/close 30741 (time 1788312060.10 total 1156.28 last 87.45) - open/close 31676 (time 1788312070.10 total 1166.28 last 93.47) - open/close 32490 (time 1788312080.14 total 1176.32 last 81.11) - open/close 33443 (time 1788312090.14 total 1186.32 last 95.28) - open/close 34479 (time 1788312100.15 total 1196.33 last 103.48) - open/close 35301 (time 1788312110.16 total 1206.34 last 82.16) - open/close 36026 (time 1788312120.16 total 1216.34 last 72.48) - open/close 36804 (time 1788312130.17 total 1226.35 last 77.73) - open/close 37590 (time 1788312140.18 total 1236.36 last 78.50) - open/close 38343 (time 1788312150.19 total 1246.37 last 75.23) - open/close 39158 (time 1788312160.20 total 1256.38 last 81.44) - open/close 40000 (time 1788312170.01 total 1266.19 last 85.81) - open/close 40894 (time 1788312180.02 total 1276.20 last 89.27) - open/close 41689 (time 1788312190.02 total 1286.20 last 79.50) - open/close 42598 (time 1788312200.03 total 1296.21 last 90.82) - open/close 43487 (time 1788312210.03 total 1306.21 last 88.90) - open/close 44299 (time 1788312220.04 total 1316.22 last 81.11) - open/close 44976 (time 1788312230.05 total 1326.23 last 67.66) - open/close 45871 (time 1788312240.06 total 1336.24 last 89.44) - open/close 46653 (time 1788312250.07 total 1346.25 last 78.09) - open/close 47567 (time 1788312260.07 total 1356.25 last 91.38) - open/close 48440 (time 1788312270.08 total 1366.26 last 87.26) - open/close 49391 (time 1788312280.08 total 1376.26 last 95.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1378.95 seconds: 35.99 ops/second - unlinked 0 (time 1788312287 ; total 0 ; last 0) - unlinked 10000 (time 1788312410 ; total 123 ; last 123) - unlinked 20000 (time 1788312524 ; total 237 ; last 114) - unlinked 30000 (time 1788312596 ; total 309 ; last 72) - unlinked 40000 (time 1788312661 ; total 374 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 439 seconds: 113.054672 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2