== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:03:32 (1786831412) start mds service on oleg251-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg251-server: detected 4 online CPUs by sysfs oleg251-server: MODOPTS_LIBCFS= oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg251-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1786831508.40 total 10.84 last 23.42) - open/close 531 (time 1786831519.05 total 21.49 last 26.01) - open/close 800 (time 1786831529.62 total 32.06 last 25.45) - open/close 1134 (time 1786831539.62 total 42.06 last 33.39) - open/close 1387 (time 1786831550.21 total 52.65 last 23.89) - open/close 1670 (time 1786831560.98 total 63.42 last 26.28) - open/close 1819 (time 1786831570.99 total 73.43 last 14.89) - open/close 1989 (time 1786831582.04 total 84.48 last 15.38) - open/close 2224 (time 1786831592.20 total 94.64 last 23.14) - open/close 2413 (time 1786831602.76 total 105.20 last 17.89) - open/close 2668 (time 1786831613.65 total 116.09 last 23.41) - open/close 2832 (time 1786831623.75 total 126.20 last 16.23) - open/close 3003 (time 1786831634.12 total 136.56 last 16.50) - open/close 3138 (time 1786831644.12 total 146.56 last 13.50) - open/close 3329 (time 1786831654.15 total 156.59 last 19.05) - open/close 3495 (time 1786831664.29 total 166.74 last 16.36) - open/close 3739 (time 1786831674.76 total 177.21 last 23.30) - open/close 3992 (time 1786831685.32 total 187.76 last 23.98) - open/close 4263 (time 1786831695.95 total 198.39 last 25.49) - open/close 4532 (time 1786831706.57 total 209.01 last 25.33) - open/close 4800 (time 1786831716.60 total 219.04 last 26.73) - open/close 5032 (time 1786831726.60 total 229.05 last 23.18) - open/close 5115 (time 1786831736.62 total 239.06 last 8.29) - open/close 5323 (time 1786831746.64 total 249.08 last 20.76) - open/close 5595 (time 1786831756.66 total 259.10 last 27.15) - open/close 5706 (time 1786831767.51 total 269.96 last 10.22) - open/close 5911 (time 1786831778.19 total 280.63 last 19.21) - open/close 6144 (time 1786831788.69 total 291.13 last 22.19) - open/close 6358 (time 1786831799.51 total 301.95 last 19.78) - open/close 6564 (time 1786831809.51 total 311.95 last 20.58) - open/close 6763 (time 1786831820.52 total 322.96 last 18.08) - open/close 7010 (time 1786831830.85 total 333.29 last 23.91) - open/close 7302 (time 1786831841.42 total 343.86 last 27.61) - open/close 7498 (time 1786831851.75 total 354.19 last 18.98) - open/close 7731 (time 1786831861.75 total 364.20 last 23.28) - open/close 7963 (time 1786831871.76 total 374.20 last 23.20) - open/close 8149 (time 1786831881.93 total 384.37 last 18.28) - open/close 8325 (time 1786831891.94 total 394.38 last 17.59) - open/close 8545 (time 1786831901.96 total 404.40 last 21.95) - open/close 8771 (time 1786831911.97 total 414.41 last 22.58) - open/close 8986 (time 1786831922.06 total 424.50 last 21.29) - open/close 9214 (time 1786831932.07 total 434.51 last 22.79) - open/close 9506 (time 1786831942.79 total 445.24 last 27.23) - open/close 9736 (time 1786831955.23 total 457.67 last 18.50) - open/close 9965 (time 1786831965.39 total 467.83 last 22.53) - open/close 10184 (time 1786831975.49 total 477.93 last 21.69) - open/close 10477 (time 1786831985.49 total 487.93 last 29.30) - open/close 10579 (time 1786831995.67 total 498.12 last 10.02) - open/close 10874 (time 1786832005.70 total 508.14 last 29.42) - open/close 11106 (time 1786832016.14 total 518.58 last 22.23) - open/close 11280 (time 1786832026.89 total 529.33 last 16.18) - open/close 11419 (time 1786832037.21 total 539.65 last 13.47) - open/close 11651 (time 1786832047.22 total 549.66 last 23.18) - open/close 11858 (time 1786832058.13 total 560.57 last 18.98) - open/close 11992 (time 1786832068.44 total 570.88 last 12.99) - open/close 12233 (time 1786832078.44 total 580.88 last 24.10) - open/close 12469 (time 1786832088.79 total 591.23 last 22.81) - open/close 12705 (time 1786832099.71 total 602.15 last 21.60) - open/close 12961 (time 1786832109.83 total 612.27 last 25.30) - open/close 13222 (time 1786832119.95 total 622.39 last 25.79) - open/close 13451 (time 1786832129.96 total 632.40 last 22.89) - open/close 13743 (time 1786832140.80 total 643.25 last 26.91) - open/close 13985 (time 1786832150.81 total 653.25 last 24.19) - open/close 14262 (time 1786832160.81 total 663.25 last 27.70) - open/close 14527 (time 1786832170.83 total 673.27 last 26.46) - open/close 14742 (time 1786832181.45 total 683.89 last 20.24) - open/close 15045 (time 1786832192.14 total 694.58 last 28.35) - open/close 15316 (time 1786832203.22 total 705.66 last 24.44) - open/close 15506 (time 1786832213.24 total 715.68 last 18.97) - open/close 15732 (time 1786832223.24 total 725.68 last 22.59) - open/close 15958 (time 1786832233.61 total 736.05 last 21.80) - open/close 16202 (time 1786832243.71 total 746.15 last 24.15) - open/close 16452 (time 1786832253.72 total 756.16 last 24.99) - open/close 16668 (time 1786832264.01 total 766.45 last 20.99) - open/close 16937 (time 1786832274.78 total 777.22 last 24.97) - open/close 17205 (time 1786832285.31 total 787.76 last 25.44) - open/close 17458 (time 1786832295.33 total 797.77 last 25.25) - open/close 17776 (time 1786832305.69 total 808.13 last 30.72) - open/close 18056 (time 1786832315.70 total 818.14 last 27.96) - open/close 18281 (time 1786832325.73 total 828.17 last 22.43) - open/close 18580 (time 1786832335.73 total 838.18 last 29.88) - open/close 18789 (time 1786832346.43 total 848.88 last 19.53) - open/close 19043 (time 1786832357.09 total 859.53 last 23.85) - open/close 19255 (time 1786832367.09 total 869.53 last 21.20) - open/close 19495 (time 1786832377.17 total 879.61 last 23.81) - open/close 19745 (time 1786832387.67 total 890.11 last 23.80) - open/close 19860 (time 1786832398.56 total 901.00 last 10.56) - open/close 20000 (time 1786832404.38 total 906.82 last 24.07) - open/close 20217 (time 1786832414.38 total 916.82 last 21.69) - open/close 20853 (time 1786832424.38 total 926.82 last 63.59) - open/close 21482 (time 1786832434.39 total 936.84 last 62.83) - open/close 22143 (time 1786832444.40 total 946.84 last 66.08) - open/close 23064 (time 1786832454.40 total 956.84 last 92.05) - open/close 24152 (time 1786832464.41 total 966.85 last 108.74) - open/close 25141 (time 1786832474.41 total 976.86 last 98.84) - open/close 26154 (time 1786832484.42 total 986.86 last 101.29) - open/close 27106 (time 1786832494.42 total 996.86 last 95.19) - open/close 27993 (time 1786832504.42 total 1006.86 last 88.65) - open/close 29061 (time 1786832514.43 total 1016.87 last 106.75) - open/close 29973 (time 1786832524.44 total 1026.88 last 91.11) - open/close 31087 (time 1786832534.44 total 1036.88 last 111.36) - open/close 32109 (time 1786832544.44 total 1046.88 last 102.19) - open/close 33040 (time 1786832554.45 total 1056.89 last 93.04) - open/close 34056 (time 1786832564.46 total 1066.90 last 101.48) - open/close 35131 (time 1786832574.47 total 1076.91 last 107.43) - open/close 36118 (time 1786832584.47 total 1086.91 last 98.65) - open/close 37199 (time 1786832594.47 total 1096.91 last 108.09) - open/close 38141 (time 1786832604.47 total 1106.92 last 94.20) - open/close 39076 (time 1786832614.48 total 1116.92 last 93.46) - open/close 40000 (time 1786832624.04 total 1126.48 last 96.67) - open/close 40838 (time 1786832634.04 total 1136.48 last 83.76) - open/close 41616 (time 1786832644.05 total 1146.49 last 77.72) - open/close 42387 (time 1786832654.07 total 1156.51 last 76.94) - open/close 43430 (time 1786832664.08 total 1166.52 last 104.23) - open/close 44329 (time 1786832674.08 total 1176.52 last 89.89) - open/close 45316 (time 1786832684.08 total 1186.52 last 98.69) - open/close 46415 (time 1786832694.09 total 1196.53 last 109.82) - open/close 47403 (time 1786832704.09 total 1206.53 last 98.75) - open/close 48481 (time 1786832714.10 total 1216.54 last 107.69) - open/close 49404 (time 1786832724.11 total 1226.55 last 92.22) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1229.14 seconds: 40.38 ops/second - unlinked 0 (time 1786832731 ; total 0 ; last 0) - unlinked 10000 (time 1786832845 ; total 114 ; last 114) - unlinked 20000 (time 1786832959 ; total 228 ; last 114) - unlinked 30000 (time 1786833023 ; total 292 ; last 64) - unlinked 40000 (time 1786833082 ; total 351 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 409 seconds: 121.347191 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules via unload_modules_local on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2