== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:10:15 (1779995415) start mds service on oleg115-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: Force libcfs to create 2 CPU partitions oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg115-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg115-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1779995519.02 total 10.00 last 25.20) - open/close 522 (time 1779995529.04 total 20.02 last 26.96) - open/close 779 (time 1779995539.98 total 30.96 last 23.48) - open/close 1046 (time 1779995549.99 total 40.97 last 26.69) - open/close 1268 (time 1779995560.72 total 51.70 last 20.69) - open/close 1514 (time 1779995570.72 total 61.70 last 24.59) - open/close 1737 (time 1779995581.45 total 72.43 last 20.79) - open/close 2010 (time 1779995591.99 total 82.97 last 25.89) - open/close 2299 (time 1779995601.99 total 92.97 last 28.89) - open/close 2558 (time 1779995612.49 total 103.47 last 24.68) - open/close 2833 (time 1779995622.49 total 113.47 last 27.50) - open/close 3052 (time 1779995633.48 total 124.46 last 19.92) - open/close 3243 (time 1779995644.18 total 135.16 last 17.85) - open/close 3417 (time 1779995655.06 total 146.04 last 15.99) - open/close 3680 (time 1779995665.07 total 156.05 last 26.27) - open/close 3943 (time 1779995675.29 total 166.27 last 25.74) - open/close 4139 (time 1779995685.75 total 176.73 last 18.74) - open/close 4374 (time 1779995695.77 total 186.75 last 23.46) - open/close 4580 (time 1779995711.68 total 202.66 last 12.95) - open/close 4832 (time 1779995722.11 total 213.09 last 24.17) - open/close 5093 (time 1779995733.05 total 224.03 last 23.86) - open/close 5227 (time 1779995743.07 total 234.05 last 13.37) - open/close 5470 (time 1779995753.08 total 244.06 last 24.28) - open/close 5711 (time 1779995763.54 total 254.52 last 23.04) - open/close 5891 (time 1779995773.84 total 264.82 last 17.47) - open/close 6088 (time 1779995786.87 total 277.85 last 15.12) - open/close 6345 (time 1779995797.21 total 288.19 last 24.87) - open/close 6531 (time 1779995807.21 total 298.19 last 18.59) - open/close 6711 (time 1779995817.73 total 308.71 last 17.11) - open/close 6887 (time 1779995827.96 total 318.94 last 17.21) - open/close 7074 (time 1779995838.85 total 329.83 last 17.17) - open/close 7189 (time 1779995848.86 total 339.84 last 11.49) - open/close 7359 (time 1779995858.88 total 349.86 last 16.97) - open/close 7542 (time 1779995869.21 total 360.19 last 17.72) - open/close 7775 (time 1779995879.62 total 370.60 last 22.38) - open/close 8042 (time 1779995890.19 total 381.17 last 25.27) - open/close 8180 (time 1779995900.20 total 391.18 last 13.78) - open/close 8329 (time 1779995910.85 total 401.83 last 14.00) - open/close 8596 (time 1779995921.10 total 412.08 last 26.05) - open/close 8873 (time 1779995931.10 total 422.08 last 27.69) - open/close 9106 (time 1779995941.75 total 432.73 last 21.88) - open/close 9233 (time 1779995951.80 total 442.78 last 12.65) - open/close 9414 (time 1779995961.80 total 452.78 last 18.10) - open/close 9613 (time 1779995972.59 total 463.57 last 18.43) - open/close 9825 (time 1779995983.54 total 474.52 last 19.37) - open/close 10000 (time 1779995993.00 total 483.98 last 18.50) - open/close 10171 (time 1779996003.00 total 493.98 last 17.09) - open/close 10279 (time 1779996013.47 total 504.45 last 10.32) - open/close 10496 (time 1779996026.43 total 517.41 last 16.74) - open/close 10733 (time 1779996037.37 total 528.35 last 21.67) - open/close 11021 (time 1779996048.44 total 539.42 last 26.00) - open/close 11246 (time 1779996059.46 total 550.44 last 20.43) - open/close 11505 (time 1779996069.49 total 560.47 last 25.83) - open/close 11721 (time 1779996079.89 total 570.87 last 20.77) - open/close 11996 (time 1779996090.37 total 581.35 last 26.24) - open/close 12173 (time 1779996100.38 total 591.36 last 17.68) - open/close 12403 (time 1779996110.40 total 601.38 last 22.95) - open/close 12642 (time 1779996121.22 total 612.20 last 22.10) - open/close 12887 (time 1779996133.81 total 624.80 last 19.45) - open/close 13087 (time 1779996144.82 total 635.80 last 18.17) - open/close 13312 (time 1779996155.56 total 646.54 last 20.97) - open/close 13465 (time 1779996165.98 total 656.96 last 14.68) - open/close 13695 (time 1779996176.89 total 667.88 last 21.07) - open/close 13824 (time 1779996186.90 total 677.88 last 12.89) - open/close 14040 (time 1779996197.19 total 688.17 last 20.99) - open/close 14351 (time 1779996207.20 total 698.18 last 31.06) - open/close 14430 (time 1779996217.97 total 708.95 last 7.34) - open/close 14569 (time 1779996229.01 total 719.99 last 12.59) - open/close 14828 (time 1779996239.74 total 730.72 last 24.14) - open/close 15101 (time 1779996250.16 total 741.14 last 26.19) - open/close 15266 (time 1779996260.53 total 751.51 last 15.92) - open/close 15478 (time 1779996271.57 total 762.55 last 19.20) - open/close 15591 (time 1779996281.57 total 772.55 last 11.30) - open/close 15774 (time 1779996291.58 total 782.56 last 18.29) - open/close 15984 (time 1779996301.59 total 792.57 last 20.98) - open/close 16160 (time 1779996311.60 total 802.58 last 17.58) - open/close 16459 (time 1779996321.62 total 812.60 last 29.84) - open/close 16594 (time 1779996331.63 total 822.61 last 13.49) - open/close 16837 (time 1779996342.15 total 833.13 last 23.10) - open/close 17060 (time 1779996353.04 total 844.02 last 20.47) - open/close 17266 (time 1779996363.13 total 854.11 last 20.42) - open/close 17555 (time 1779996373.15 total 864.13 last 28.83) - open/close 17763 (time 1779996383.24 total 874.22 last 20.63) - open/close 17850 (time 1779996393.54 total 884.52 last 8.44) - open/close 18070 (time 1779996404.31 total 895.29 last 20.43) - open/close 18221 (time 1779996414.32 total 905.30 last 15.09) - open/close 18395 (time 1779996424.34 total 915.32 last 17.37) - open/close 18600 (time 1779996434.35 total 925.33 last 20.47) - open/close 18728 (time 1779996444.37 total 935.35 last 12.77) - open/close 18873 (time 1779996454.39 total 945.37 last 14.47) - open/close 18958 (time 1779996464.40 total 955.38 last 8.49) - open/close 19111 (time 1779996474.90 total 965.88 last 14.57) - open/close 19368 (time 1779996484.91 total 975.89 last 25.68) - open/close 19569 (time 1779996494.92 total 985.90 last 20.08) - open/close 20000 (time 1779996501.62 total 992.60 last 64.30) - open/close 20884 (time 1779996511.62 total 1002.60 last 88.37) - open/close 21745 (time 1779996521.62 total 1012.60 last 86.10) - open/close 22613 (time 1779996531.63 total 1022.61 last 86.78) - open/close 23556 (time 1779996541.63 total 1032.61 last 94.24) - open/close 24394 (time 1779996551.64 total 1042.62 last 83.76) - open/close 25200 (time 1779996561.64 total 1052.62 last 80.58) - open/close 26020 (time 1779996571.64 total 1062.62 last 82.00) - open/close 26868 (time 1779996581.64 total 1072.62 last 84.77) - open/close 27756 (time 1779996591.64 total 1082.62 last 88.78) - open/close 28709 (time 1779996601.65 total 1092.63 last 95.26) - open/close 29828 (time 1779996611.65 total 1102.63 last 111.89) - open/close 30595 (time 1779996621.65 total 1112.63 last 76.66) - open/close 31491 (time 1779996631.66 total 1122.64 last 89.57) - open/close 32161 (time 1779996641.68 total 1132.66 last 66.86) - open/close 32916 (time 1779996651.68 total 1142.66 last 75.48) - open/close 33733 (time 1779996661.68 total 1152.66 last 81.67) - open/close 34565 (time 1779996671.69 total 1162.67 last 83.12) - open/close 35600 (time 1779996681.70 total 1172.68 last 103.40) - open/close 36454 (time 1779996691.71 total 1182.69 last 85.37) - open/close 37256 (time 1779996701.72 total 1192.70 last 80.05) - open/close 38123 (time 1779996711.73 total 1202.71 last 86.69) - open/close 39132 (time 1779996721.73 total 1212.71 last 100.87) - open/close 39966 (time 1779996731.74 total 1222.72 last 83.34) - open/close 40883 (time 1779996741.74 total 1232.72 last 91.65) - open/close 41776 (time 1779996751.75 total 1242.73 last 89.24) - open/close 42712 (time 1779996761.75 total 1252.73 last 93.59) - open/close 43685 (time 1779996771.76 total 1262.74 last 97.23) - open/close 44563 (time 1779996781.76 total 1272.74 last 87.77) - open/close 45312 (time 1779996791.76 total 1282.74 last 74.90) - open/close 46198 (time 1779996801.78 total 1292.76 last 88.44) - open/close 46934 (time 1779996811.78 total 1302.76 last 73.59) - open/close 47706 (time 1779996821.80 total 1312.78 last 77.06) - open/close 48543 (time 1779996831.80 total 1322.78 last 83.66) - open/close 49424 (time 1779996841.81 total 1332.79 last 88.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1335.55 seconds: 37.16 ops/second - unlinked 0 (time 1779996850 ; total 0 ; last 0) - unlinked 10000 (time 1779996972 ; total 122 ; last 122) - unlinked 20000 (time 1779997097 ; total 247 ; last 125) - unlinked 30000 (time 1779997169 ; total 319 ; last 72) - unlinked 40000 (time 1779997240 ; total 390 ; last 71) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 454 seconds: 109.319382 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2