== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:57:04 (1763348224) start mds service on oleg414-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg414-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1763348296.94 total 10.97 last 10.85) - open/close 234 (time 1763348307.62 total 21.65 last 10.77) - open/close 376 (time 1763348317.99 total 32.02 last 13.69) - open/close 547 (time 1763348329.00 total 43.03 last 15.53) - open/close 777 (time 1763348339.01 total 53.04 last 22.98) - open/close 982 (time 1763348349.86 total 63.89 last 18.90) - open/close 1148 (time 1763348360.80 total 74.83 last 15.17) - open/close 1328 (time 1763348370.92 total 84.95 last 17.78) - open/close 1476 (time 1763348381.47 total 95.51 last 14.02) - open/close 1767 (time 1763348391.49 total 105.52 last 29.06) - open/close 1989 (time 1763348402.10 total 116.13 last 20.93) - open/close 2184 (time 1763348412.34 total 126.37 last 19.03) - open/close 2360 (time 1763348422.35 total 136.38 last 17.59) - open/close 2447 (time 1763348432.82 total 146.85 last 8.31) - open/close 2680 (time 1763348443.70 total 157.73 last 21.42) - open/close 2814 (time 1763348453.80 total 167.83 last 13.26) - open/close 2946 (time 1763348463.90 total 177.94 last 13.06) - open/close 3193 (time 1763348474.73 total 188.76 last 22.81) - open/close 3458 (time 1763348484.73 total 198.77 last 26.49) - open/close 3581 (time 1763348495.72 total 209.75 last 11.20) - open/close 3886 (time 1763348506.02 total 220.05 last 29.60) - open/close 3974 (time 1763348516.44 total 230.47 last 8.44) - open/close 4153 (time 1763348526.46 total 240.49 last 17.87) - open/close 4338 (time 1763348536.67 total 250.70 last 18.12) - open/close 4504 (time 1763348546.68 total 260.71 last 16.59) - open/close 4684 (time 1763348556.69 total 270.73 last 17.97) - open/close 4854 (time 1763348567.20 total 281.23 last 16.18) - open/close 5037 (time 1763348577.95 total 291.98 last 17.02) - open/close 5092 (time 1763348588.01 total 302.05 last 5.47) - open/close 5236 (time 1763348598.02 total 312.05 last 14.39) - open/close 5351 (time 1763348608.93 total 322.96 last 10.54) - open/close 5562 (time 1763348619.68 total 333.71 last 19.62) - open/close 5731 (time 1763348630.45 total 344.48 last 15.70) - open/close 5954 (time 1763348640.45 total 354.48 last 22.29) - open/close 6146 (time 1763348651.06 total 365.09 last 18.10) - open/close 6266 (time 1763348661.06 total 375.09 last 11.99) - open/close 6436 (time 1763348671.84 total 385.87 last 15.78) - open/close 6614 (time 1763348681.84 total 395.87 last 17.79) - open/close 6840 (time 1763348692.70 total 406.74 last 20.81) - open/close 7004 (time 1763348703.33 total 417.36 last 15.44) - open/close 7132 (time 1763348713.52 total 427.55 last 12.56) - open/close 7275 (time 1763348723.52 total 437.55 last 14.30) - open/close 7481 (time 1763348733.87 total 447.90 last 19.91) - open/close 7692 (time 1763348744.49 total 458.52 last 19.87) - open/close 7901 (time 1763348754.78 total 468.81 last 20.31) - open/close 8119 (time 1763348765.42 total 479.45 last 20.50) - open/close 8363 (time 1763348775.43 total 489.46 last 24.37) - open/close 8483 (time 1763348785.44 total 499.47 last 11.99) - open/close 8595 (time 1763348796.46 total 510.49 last 10.16) - open/close 8790 (time 1763348806.81 total 520.84 last 18.83) - open/close 9004 (time 1763348817.18 total 531.22 last 20.63) - open/close 9223 (time 1763348828.00 total 542.04 last 20.24) - open/close 9461 (time 1763348839.01 total 553.04 last 21.63) - open/close 9673 (time 1763348850.04 total 564.07 last 19.22) - open/close 9858 (time 1763348860.26 total 574.29 last 18.10) - open/close 9989 (time 1763348870.27 total 584.30 last 13.09) - open/close 10161 (time 1763348880.36 total 594.39 last 17.04) - open/close 10399 (time 1763348891.32 total 605.35 last 21.71) - open/close 10660 (time 1763348902.00 total 616.03 last 24.45) - open/close 10939 (time 1763348912.01 total 626.04 last 27.87) - open/close 11217 (time 1763348922.88 total 636.91 last 25.57) - open/close 11420 (time 1763348932.88 total 646.91 last 20.29) - open/close 11627 (time 1763348943.61 total 657.65 last 19.29) - open/close 11848 (time 1763348953.62 total 667.65 last 22.09) - open/close 12054 (time 1763348964.13 total 678.17 last 19.59) - open/close 12775 (time 1763348974.14 total 688.18 last 72.03) - open/close 13554 (time 1763348984.15 total 698.18 last 77.84) - open/close 14403 (time 1763348994.15 total 708.19 last 84.87) - open/close 15210 (time 1763349004.17 total 718.20 last 80.61) - open/close 15990 (time 1763349014.17 total 728.20 last 77.98) - open/close 16778 (time 1763349024.17 total 738.21 last 78.75) - open/close 17505 (time 1763349034.18 total 748.21 last 72.69) - open/close 18076 (time 1763349044.18 total 758.22 last 57.05) - open/close 18818 (time 1763349054.21 total 768.24 last 74.04) - open/close 19606 (time 1763349064.21 total 778.25 last 78.74) - open/close 20000 (time 1763349068.89 total 782.92 last 84.25) - open/close 20846 (time 1763349078.90 total 792.93 last 84.55) - open/close 21582 (time 1763349088.90 total 802.93 last 73.58) - open/close 22317 (time 1763349098.90 total 812.93 last 73.49) - open/close 22957 (time 1763349108.90 total 822.94 last 63.98) - open/close 23607 (time 1763349118.93 total 832.96 last 64.81) - open/close 24277 (time 1763349128.94 total 842.97 last 66.97) - open/close 25087 (time 1763349138.95 total 852.98 last 80.91) - open/close 25871 (time 1763349148.95 total 862.98 last 78.37) - open/close 26487 (time 1763349158.95 total 872.99 last 61.58) - open/close 27208 (time 1763349168.97 total 883.00 last 72.01) - open/close 27959 (time 1763349178.97 total 893.00 last 75.07) - open/close 28797 (time 1763349188.98 total 903.01 last 83.75) - open/close 29651 (time 1763349198.98 total 913.01 last 85.39) - open/close 30000 (time 1763349202.91 total 916.95 last 88.69) - open/close 30768 (time 1763349212.92 total 926.95 last 76.75) - open/close 31512 (time 1763349222.92 total 936.96 last 74.38) - open/close 32200 (time 1763349232.93 total 946.96 last 68.74) - open/close 32900 (time 1763349242.93 total 956.97 last 69.99) - open/close 33619 (time 1763349252.94 total 966.98 last 71.83) - open/close 34398 (time 1763349262.96 total 976.99 last 77.79) - open/close 35254 (time 1763349272.96 total 987.00 last 85.54) - open/close 35982 (time 1763349282.97 total 997.00 last 72.79) - open/close 36743 (time 1763349292.98 total 1007.01 last 75.99) - open/close 37561 (time 1763349302.99 total 1017.02 last 81.74) - open/close 38408 (time 1763349312.99 total 1027.02 last 84.69) - open/close 39200 (time 1763349322.99 total 1037.02 last 79.17) - open/close 39989 (time 1763349333.00 total 1047.03 last 78.87) - open/close 40854 (time 1763349343.00 total 1057.03 last 86.49) - open/close 41609 (time 1763349353.00 total 1067.04 last 75.45) - open/close 42484 (time 1763349363.01 total 1077.04 last 87.45) - open/close 43320 (time 1763349373.01 total 1087.05 last 83.56) - open/close 44113 (time 1763349383.03 total 1097.06 last 79.17) - open/close 44882 (time 1763349393.03 total 1107.07 last 76.87) - open/close 45703 (time 1763349403.04 total 1117.08 last 82.03) - open/close 46469 (time 1763349413.05 total 1127.08 last 76.56) - open/close 47250 (time 1763349423.06 total 1137.09 last 78.04) - open/close 47922 (time 1763349433.06 total 1147.09 last 67.19) - open/close 48682 (time 1763349443.06 total 1157.10 last 75.94) - open/close 49383 (time 1763349453.07 total 1167.10 last 70.07) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1170.92 seconds: 42.39 ops/second - unlinked 0 (time 1763349461 ; total 0 ; last 0) - unlinked 10000 (time 1763349591 ; total 130 ; last 130) - unlinked 20000 (time 1763349723 ; total 262 ; last 132) - unlinked 30000 (time 1763349858 ; total 397 ; last 135) - unlinked 40000 (time 1763349994 ; total 533 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 661 seconds: 75.086235 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg414-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2