== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:05:43 (1773612343) start mds service on oleg360-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg360-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg360-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg360-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 249 (time 1773612437.72 total 10.81 last 23.03) - open/close 486 (time 1773612447.83 total 20.93 last 23.43) - open/close 704 (time 1773612458.46 total 31.55 last 20.52) - open/close 921 (time 1773612468.50 total 41.60 last 21.61) - open/close 1198 (time 1773612479.31 total 52.41 last 25.62) - open/close 1480 (time 1773612490.39 total 63.49 last 25.45) - open/close 1706 (time 1773612500.40 total 73.50 last 22.58) - open/close 1930 (time 1773612510.40 total 83.50 last 22.40) - open/close 2054 (time 1773612520.42 total 93.51 last 12.38) - open/close 2187 (time 1773612531.34 total 104.43 last 12.18) - open/close 2435 (time 1773612541.96 total 115.05 last 23.35) - open/close 2634 (time 1773612552.11 total 125.20 last 19.61) - open/close 2878 (time 1773612562.91 total 136.01 last 22.58) - open/close 3118 (time 1773612573.16 total 146.26 last 23.41) - open/close 3421 (time 1773612583.70 total 156.79 last 28.75) - open/close 3632 (time 1773612593.82 total 166.92 last 20.84) - open/close 3811 (time 1773612603.85 total 176.95 last 17.85) - open/close 4075 (time 1773612614.43 total 187.52 last 24.97) - open/close 4281 (time 1773612624.43 total 197.53 last 20.59) - open/close 4541 (time 1773612635.27 total 208.37 last 23.98) - open/close 4802 (time 1773612645.72 total 218.81 last 24.99) - open/close 5063 (time 1773612655.72 total 228.82 last 26.09) - open/close 5294 (time 1773612665.73 total 238.83 last 23.08) - open/close 5538 (time 1773612676.06 total 249.16 last 23.61) - open/close 5804 (time 1773612686.99 total 260.09 last 24.34) - open/close 6085 (time 1773612697.01 total 270.10 last 28.06) - open/close 6366 (time 1773612707.02 total 280.11 last 28.06) - open/close 6614 (time 1773612717.42 total 290.52 last 23.84) - open/close 6890 (time 1773612728.41 total 301.51 last 25.12) - open/close 7105 (time 1773612738.42 total 311.51 last 21.49) - open/close 7339 (time 1773612748.49 total 321.59 last 23.23) - open/close 7639 (time 1773612759.32 total 332.41 last 27.72) - open/close 7893 (time 1773612769.32 total 342.41 last 25.39) - open/close 8225 (time 1773612779.32 total 352.41 last 33.20) - open/close 8358 (time 1773612790.05 total 363.15 last 12.39) - open/close 8585 (time 1773612800.14 total 373.24 last 22.49) - open/close 8823 (time 1773612810.32 total 383.42 last 23.39) - open/close 9095 (time 1773612821.33 total 394.43 last 24.70) - open/close 9302 (time 1773612831.46 total 404.55 last 20.45) - open/close 9552 (time 1773612841.74 total 414.83 last 24.31) - open/close 9840 (time 1773612851.92 total 425.02 last 28.28) - open/close 10000 (time 1773612858.36 total 431.46 last 24.84) - open/close 10244 (time 1773612868.37 total 441.47 last 24.38) - open/close 10386 (time 1773612878.86 total 451.95 last 13.54) - open/close 10631 (time 1773612888.86 total 461.95 last 24.50) - open/close 10869 (time 1773612898.86 total 471.96 last 23.79) - open/close 11134 (time 1773612909.03 total 482.12 last 26.07) - open/close 11387 (time 1773612920.02 total 493.11 last 23.02) - open/close 11637 (time 1773612932.76 total 505.85 last 19.63) - open/close 11860 (time 1773612943.55 total 516.64 last 20.66) - open/close 12024 (time 1773612955.11 total 528.21 last 14.18) - open/close 12271 (time 1773612965.58 total 538.67 last 23.60) - open/close 12510 (time 1773612976.23 total 549.32 last 22.45) - open/close 12739 (time 1773612986.23 total 559.32 last 22.90) - open/close 12976 (time 1773612996.24 total 569.34 last 23.66) - open/close 13213 (time 1773613006.25 total 579.35 last 23.68) - open/close 13465 (time 1773613016.26 total 589.35 last 25.19) - open/close 13738 (time 1773613026.90 total 600.00 last 25.65) - open/close 13996 (time 1773613037.34 total 610.43 last 24.72) - open/close 14186 (time 1773613047.45 total 620.54 last 18.79) - open/close 14483 (time 1773613058.07 total 631.16 last 27.97) - open/close 14724 (time 1773613068.69 total 641.78 last 22.70) - open/close 14948 (time 1773613078.93 total 652.03 last 21.87) - open/close 15206 (time 1773613088.93 total 662.03 last 25.79) - open/close 15458 (time 1773613099.23 total 672.32 last 24.47) - open/close 15707 (time 1773613109.25 total 682.34 last 24.85) - open/close 15934 (time 1773613119.56 total 692.66 last 22.02) - open/close 16150 (time 1773613130.58 total 703.68 last 19.60) - open/close 16384 (time 1773613140.58 total 713.68 last 23.40) - open/close 16628 (time 1773613150.61 total 723.70 last 24.35) - open/close 16855 (time 1773613161.24 total 734.34 last 21.34) - open/close 17126 (time 1773613171.61 total 744.70 last 26.14) - open/close 17377 (time 1773613182.56 total 755.65 last 22.93) - open/close 17617 (time 1773613193.17 total 766.26 last 22.62) - open/close 17832 (time 1773613203.42 total 776.51 last 20.97) - open/close 18096 (time 1773613213.42 total 786.52 last 26.39) - open/close 18354 (time 1773613224.42 total 797.51 last 23.47) - open/close 18583 (time 1773613235.15 total 808.24 last 21.34) - open/close 18764 (time 1773613246.11 total 819.21 last 16.51) - open/close 18895 (time 1773613256.12 total 829.21 last 13.09) - open/close 19071 (time 1773613266.98 total 840.07 last 16.20) - open/close 19575 (time 1773613276.98 total 850.08 last 50.37) - open/close 20000 (time 1773613280.44 total 853.54 last 122.84) - open/close 21185 (time 1773613290.45 total 863.54 last 118.45) - open/close 22238 (time 1773613300.45 total 873.55 last 105.25) - open/close 23380 (time 1773613310.46 total 883.55 last 114.14) - open/close 24523 (time 1773613320.47 total 893.56 last 114.21) - open/close 25635 (time 1773613330.48 total 903.57 last 111.08) - open/close 26664 (time 1773613340.48 total 913.57 last 102.88) - open/close 27811 (time 1773613350.50 total 923.59 last 114.48) - open/close 28719 (time 1773613360.50 total 933.60 last 90.78) - open/close 29262 (time 1773613370.52 total 943.61 last 54.20) - open/close 29855 (time 1773613380.53 total 953.62 last 59.25) - open/close 30000 (time 1773613384.03 total 957.12 last 41.42) - open/close 30631 (time 1773613394.04 total 967.13 last 63.03) - open/close 31469 (time 1773613404.04 total 977.14 last 83.77) - open/close 32278 (time 1773613414.05 total 987.15 last 80.80) - open/close 33232 (time 1773613424.06 total 997.15 last 95.38) - open/close 34203 (time 1773613434.06 total 1007.16 last 97.06) - open/close 35201 (time 1773613444.06 total 1017.16 last 99.78) - open/close 36263 (time 1773613454.06 total 1027.16 last 106.19) - open/close 37148 (time 1773613464.07 total 1037.17 last 88.40) - open/close 38117 (time 1773613474.09 total 1047.19 last 96.74) - open/close 39085 (time 1773613484.09 total 1057.19 last 96.80) - open/close 40000 (time 1773613493.08 total 1066.17 last 101.84) - open/close 40996 (time 1773613503.08 total 1076.17 last 99.56) - open/close 41584 (time 1773613513.08 total 1086.18 last 58.77) - open/close 42243 (time 1773613523.11 total 1096.20 last 65.74) - open/close 42878 (time 1773613533.11 total 1106.21 last 63.48) - open/close 43604 (time 1773613543.11 total 1116.21 last 72.58) - open/close 44266 (time 1773613553.12 total 1126.22 last 66.15) - open/close 45142 (time 1773613563.15 total 1136.25 last 87.33) - open/close 45995 (time 1773613573.17 total 1146.26 last 85.17) - open/close 47059 (time 1773613583.18 total 1156.27 last 106.31) - open/close 48034 (time 1773613593.21 total 1166.31 last 97.16) - open/close 49038 (time 1773613603.26 total 1176.35 last 99.92) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1182.90 seconds: 41.96 ops/second - unlinked 0 (time 1773613614 ; total 0 ; last 0) - unlinked 10000 (time 1773613718 ; total 104 ; last 104) - unlinked 20000 (time 1773613816 ; total 202 ; last 98) - unlinked 30000 (time 1773613876 ; total 262 ; last 60) - unlinked 40000 (time 1773613936 ; total 322 ; last 60) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 382 seconds: 129.926697 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2