== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:31:28 (1787398288) start mds service on oleg324-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg324-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg324-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg324-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 255 (time 1787398388.11 total 11.34 last 22.49) - open/close 477 (time 1787398399.14 total 22.36 last 20.13) - open/close 768 (time 1787398409.14 total 32.37 last 29.09) - open/close 997 (time 1787398419.47 total 42.70 last 22.17) - open/close 1260 (time 1787398430.06 total 53.29 last 24.83) - open/close 1549 (time 1787398440.21 total 63.44 last 28.47) - open/close 1842 (time 1787398450.23 total 73.45 last 29.26) - open/close 2093 (time 1787398460.76 total 83.98 last 23.84) - open/close 2391 (time 1787398471.63 total 94.85 last 27.42) - open/close 2598 (time 1787398481.64 total 104.86 last 20.68) - open/close 2835 (time 1787398491.66 total 114.89 last 23.63) - open/close 3159 (time 1787398502.62 total 125.85 last 29.57) - open/close 3401 (time 1787398513.43 total 136.65 last 22.39) - open/close 3654 (time 1787398524.44 total 147.66 last 22.98) - open/close 3924 (time 1787398534.61 total 157.83 last 26.55) - open/close 4133 (time 1787398544.61 total 167.84 last 20.89) - open/close 4333 (time 1787398554.99 total 178.21 last 19.28) - open/close 4636 (time 1787398564.99 total 188.22 last 30.29) - open/close 4880 (time 1787398575.71 total 198.94 last 22.75) - open/close 5133 (time 1787398586.43 total 209.66 last 23.61) - open/close 5306 (time 1787398597.45 total 220.68 last 15.69) - open/close 5612 (time 1787398607.46 total 230.68 last 30.59) - open/close 5849 (time 1787398618.38 total 241.60 last 21.70) - open/close 6113 (time 1787398628.44 total 251.67 last 26.23) - open/close 6395 (time 1787398639.34 total 262.56 last 25.89) - open/close 6589 (time 1787398649.80 total 273.03 last 18.54) - open/close 6755 (time 1787398660.04 total 283.27 last 16.21) - open/close 6851 (time 1787398670.92 total 294.15 last 8.83) - open/close 7063 (time 1787398681.87 total 305.09 last 19.37) - open/close 7311 (time 1787398692.23 total 315.46 last 23.93) - open/close 7596 (time 1787398702.37 total 325.60 last 28.11) - open/close 7958 (time 1787398712.37 total 335.60 last 36.20) - open/close 8186 (time 1787398722.37 total 345.60 last 22.79) - open/close 8429 (time 1787398732.38 total 355.61 last 24.29) - open/close 8708 (time 1787398743.07 total 366.30 last 26.09) - open/close 8930 (time 1787398753.09 total 376.32 last 22.16) - open/close 9228 (time 1787398763.09 total 386.32 last 29.80) - open/close 9454 (time 1787398773.10 total 396.32 last 22.59) - open/close 9757 (time 1787398783.10 total 406.32 last 30.30) - open/close 9983 (time 1787398793.38 total 416.61 last 21.98) - open/close 10000 (time 1787398795.78 total 419.01 last 7.08) - open/close 10213 (time 1787398806.01 total 429.24 last 20.82) - open/close 10504 (time 1787398816.03 total 439.26 last 29.05) - open/close 10766 (time 1787398826.04 total 449.26 last 26.18) - open/close 11016 (time 1787398836.12 total 459.35 last 24.80) - open/close 11294 (time 1787398846.67 total 469.90 last 26.34) - open/close 11527 (time 1787398856.78 total 480.00 last 23.07) - open/close 11806 (time 1787398866.81 total 490.04 last 27.80) - open/close 12068 (time 1787398877.27 total 500.49 last 25.06) - open/close 12239 (time 1787398887.46 total 510.69 last 16.77) - open/close 12478 (time 1787398897.47 total 520.69 last 23.89) - open/close 12717 (time 1787398907.47 total 530.69 last 23.90) - open/close 12957 (time 1787398917.48 total 540.71 last 23.96) - open/close 13235 (time 1787398928.20 total 551.43 last 25.94) - open/close 13519 (time 1787398938.27 total 561.50 last 28.20) - open/close 13801 (time 1787398948.28 total 571.50 last 28.19) - open/close 14060 (time 1787398959.00 total 582.22 last 24.16) - open/close 14309 (time 1787398969.00 total 592.22 last 24.90) - open/close 14559 (time 1787398979.01 total 602.23 last 24.98) - open/close 14802 (time 1787398989.19 total 612.42 last 23.85) - open/close 15076 (time 1787398999.20 total 622.43 last 27.39) - open/close 15359 (time 1787399009.21 total 632.44 last 28.26) - open/close 15612 (time 1787399019.31 total 642.53 last 25.07) - open/close 15818 (time 1787399032.28 total 655.50 last 15.88) - open/close 16035 (time 1787399042.29 total 665.52 last 21.67) - open/close 16293 (time 1787399052.58 total 675.81 last 25.07) - open/close 16562 (time 1787399063.30 total 686.53 last 25.10) - open/close 16814 (time 1787399073.52 total 696.74 last 24.66) - open/close 17084 (time 1787399083.86 total 707.08 last 26.11) - open/close 17293 (time 1787399093.88 total 717.11 last 20.85) - open/close 17400 (time 1787399103.90 total 727.13 last 10.68) - open/close 17628 (time 1787399114.48 total 737.71 last 21.55) - open/close 17794 (time 1787399125.03 total 748.26 last 15.74) - open/close 17975 (time 1787399135.78 total 759.00 last 16.85) - open/close 18276 (time 1787399145.80 total 769.02 last 30.04) - open/close 18497 (time 1787399156.76 total 779.98 last 20.16) - open/close 18764 (time 1787399167.00 total 790.22 last 26.08) - open/close 18970 (time 1787399178.52 total 801.74 last 17.89) - open/close 19129 (time 1787399189.73 total 812.95 last 14.18) - open/close 19401 (time 1787399199.75 total 822.98 last 27.14) - open/close 19679 (time 1787399209.75 total 832.98 last 27.79) - open/close 19944 (time 1787399220.46 total 843.68 last 24.76) - open/close 20000 (time 1787399222.68 total 845.91 last 25.15) - open/close 20206 (time 1787399233.24 total 856.47 last 19.51) - open/close 20502 (time 1787399243.24 total 866.47 last 29.59) - open/close 20798 (time 1787399254.12 total 877.35 last 27.21) - open/close 21085 (time 1787399264.14 total 887.37 last 28.64) - open/close 21267 (time 1787399274.85 total 898.08 last 16.99) - open/close 21552 (time 1787399284.86 total 908.09 last 28.46) - open/close 21779 (time 1787399294.87 total 918.10 last 22.69) - open/close 22577 (time 1787399304.88 total 928.11 last 79.70) - open/close 23417 (time 1787399314.88 total 938.11 last 83.99) - open/close 24304 (time 1787399324.89 total 948.12 last 88.66) - open/close 25129 (time 1787399334.90 total 958.12 last 82.45) - open/close 26002 (time 1787399344.90 total 968.12 last 87.30) - open/close 26817 (time 1787399354.91 total 978.14 last 81.37) - open/close 27590 (time 1787399364.92 total 988.14 last 77.26) - open/close 28323 (time 1787399374.92 total 998.15 last 73.26) - open/close 29149 (time 1787399384.92 total 1008.15 last 82.59) - open/close 29978 (time 1787399394.93 total 1018.16 last 82.85) - open/close 30794 (time 1787399404.93 total 1028.16 last 81.59) - open/close 31709 (time 1787399414.93 total 1038.16 last 91.49) - open/close 32569 (time 1787399424.94 total 1048.17 last 85.91) - open/close 33407 (time 1787399434.95 total 1058.17 last 83.77) - open/close 34275 (time 1787399444.95 total 1068.17 last 86.80) - open/close 35192 (time 1787399454.96 total 1078.18 last 91.60) - open/close 36037 (time 1787399464.98 total 1088.21 last 84.33) - open/close 36838 (time 1787399474.98 total 1098.21 last 80.06) - open/close 37730 (time 1787399485.00 total 1108.23 last 89.03) - open/close 38633 (time 1787399495.01 total 1118.24 last 90.24) - open/close 39493 (time 1787399505.01 total 1128.24 last 85.99) - open/close 40000 (time 1787399511.18 total 1134.40 last 82.23) - open/close 40860 (time 1787399521.20 total 1144.43 last 85.79) - open/close 41668 (time 1787399531.21 total 1154.44 last 80.71) - open/close 42360 (time 1787399541.22 total 1164.45 last 69.12) - open/close 43164 (time 1787399551.24 total 1174.46 last 80.28) - open/close 44059 (time 1787399561.26 total 1184.49 last 89.30) - open/close 44888 (time 1787399571.26 total 1194.49 last 82.88) - open/close 45696 (time 1787399581.27 total 1204.49 last 80.76) - open/close 46512 (time 1787399591.28 total 1214.50 last 81.52) - open/close 47298 (time 1787399601.28 total 1224.51 last 78.56) - open/close 48171 (time 1787399611.29 total 1234.52 last 87.20) - open/close 49076 (time 1787399621.31 total 1244.53 last 90.37) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1251.02 seconds: 39.67 ops/second - unlinked 0 (time 1787399632 ; total 0 ; last 0) - unlinked 10000 (time 1787399755 ; total 123 ; last 123) - unlinked 20000 (time 1787399878 ; total 246 ; last 123) - unlinked 30000 (time 1787399999 ; total 367 ; last 121) - unlinked 40000 (time 1787400115 ; total 483 ; last 116) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 574 seconds: 86.465157 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2