== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:05:27 (1786817127) start mds service on oleg614-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg614-server' oleg614-server: oleg614-server.virtnet: executing load_modules_local oleg614-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg614-server: Loading modules from /home/green/git/lustre-release/lustre oleg614-server: detected 4 online CPUs by sysfs oleg614-server: MODOPTS_LIBCFS= oleg614-server: Force libcfs to create 2 CPU partitions oleg614-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg614-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg614-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg614-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg614-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg614-client.virtnet: -o user_xattr,flock 192.168.206.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1786817219.67 total 10.10 last 25.54) - open/close 496 (time 1786817230.24 total 20.67 last 22.52) - open/close 703 (time 1786817240.35 total 30.78 last 20.47) - open/close 913 (time 1786817250.36 total 40.79 last 20.98) - open/close 1098 (time 1786817261.08 total 51.52 last 17.25) - open/close 1340 (time 1786817271.71 total 62.14 last 22.78) - open/close 1585 (time 1786817282.34 total 72.77 last 23.04) - open/close 1802 (time 1786817292.35 total 82.78 last 21.68) - open/close 2037 (time 1786817302.80 total 93.24 last 22.48) - open/close 2295 (time 1786817312.81 total 103.24 last 25.79) - open/close 2533 (time 1786817323.73 total 114.16 last 21.79) - open/close 2741 (time 1786817333.74 total 124.17 last 20.79) - open/close 2992 (time 1786817343.77 total 134.21 last 25.01) - open/close 3243 (time 1786817354.72 total 145.15 last 22.94) - open/close 3481 (time 1786817364.73 total 155.16 last 23.78) - open/close 3717 (time 1786817375.45 total 165.89 last 22.00) - open/close 3953 (time 1786817385.51 total 175.94 last 23.48) - open/close 4184 (time 1786817395.54 total 185.97 last 23.03) - open/close 4457 (time 1786817406.51 total 196.95 last 24.87) - open/close 4648 (time 1786817416.53 total 206.96 last 19.07) - open/close 4814 (time 1786817426.54 total 216.97 last 16.58) - open/close 4895 (time 1786817436.55 total 226.98 last 8.09) - open/close 5122 (time 1786817446.55 total 236.99 last 22.69) - open/close 5333 (time 1786817456.57 total 247.00 last 21.06) - open/close 5614 (time 1786817466.58 total 257.01 last 28.08) - open/close 5884 (time 1786817477.34 total 267.77 last 25.10) - open/close 6157 (time 1786817487.39 total 277.82 last 27.16) - open/close 6415 (time 1786817497.49 total 287.92 last 25.54) - open/close 6622 (time 1786817508.50 total 298.93 last 18.80) - open/close 6839 (time 1786817518.53 total 308.96 last 21.65) - open/close 7102 (time 1786817529.50 total 319.93 last 23.96) - open/close 7304 (time 1786817539.52 total 329.95 last 20.16) - open/close 7502 (time 1786817549.53 total 339.96 last 19.79) - open/close 7783 (time 1786817559.53 total 349.96 last 28.09) - open/close 8060 (time 1786817570.16 total 360.59 last 26.05) - open/close 8306 (time 1786817581.10 total 371.53 last 22.50) - open/close 8604 (time 1786817591.10 total 381.54 last 29.77) - open/close 8812 (time 1786817601.76 total 392.19 last 19.52) - open/close 9026 (time 1786817612.06 total 402.49 last 20.78) - open/close 9289 (time 1786817622.37 total 412.80 last 25.51) - open/close 9495 (time 1786817634.40 total 424.83 last 17.12) - open/close 9628 (time 1786817645.19 total 435.62 last 12.33) - open/close 9814 (time 1786817655.89 total 446.32 last 17.38) - open/close 10000 (time 1786817663.08 total 453.51 last 25.88) - open/close 10208 (time 1786817673.10 total 463.53 last 20.75) - open/close 10521 (time 1786817683.81 total 474.24 last 29.24) - open/close 10804 (time 1786817693.81 total 484.24 last 28.29) - open/close 11016 (time 1786817703.83 total 494.27 last 21.15) - open/close 11280 (time 1786817714.09 total 504.52 last 25.75) - open/close 11357 (time 1786817724.13 total 514.57 last 7.66) - open/close 11455 (time 1786817734.55 total 524.98 last 9.41) - open/close 11564 (time 1786817745.39 total 535.82 last 10.06) - open/close 11775 (time 1786817755.40 total 545.83 last 21.08) - open/close 12031 (time 1786817766.38 total 556.81 last 23.32) - open/close 12264 (time 1786817776.38 total 566.82 last 23.28) - open/close 12528 (time 1786817786.94 total 577.37 last 25.02) - open/close 12809 (time 1786817797.93 total 588.36 last 25.56) - open/close 13112 (time 1786817807.93 total 598.36 last 30.29) - open/close 13331 (time 1786817818.46 total 608.89 last 20.80) - open/close 13592 (time 1786817829.49 total 619.92 last 23.67) - open/close 13845 (time 1786817839.51 total 629.94 last 25.24) - open/close 14143 (time 1786817849.51 total 639.95 last 29.80) - open/close 14390 (time 1786817860.19 total 650.62 last 23.14) - open/close 14622 (time 1786817870.19 total 660.62 last 23.19) - open/close 14848 (time 1786817880.41 total 670.84 last 22.12) - open/close 15132 (time 1786817890.42 total 680.85 last 28.37) - open/close 15383 (time 1786817900.98 total 691.41 last 23.78) - open/close 15652 (time 1786817911.91 total 702.34 last 24.61) - open/close 15965 (time 1786817921.91 total 712.34 last 31.29) - open/close 16148 (time 1786817932.33 total 722.76 last 17.56) - open/close 16440 (time 1786817943.27 total 733.70 last 26.70) - open/close 16685 (time 1786817953.28 total 743.72 last 24.46) - open/close 16930 (time 1786817963.29 total 753.72 last 24.49) - open/close 17166 (time 1786817973.91 total 764.34 last 22.22) - open/close 17402 (time 1786817984.68 total 775.12 last 21.90) - open/close 17609 (time 1786817994.94 total 785.37 last 20.19) - open/close 17825 (time 1786818004.94 total 795.37 last 21.59) - open/close 17989 (time 1786818015.11 total 805.54 last 16.12) - open/close 18137 (time 1786818025.63 total 816.06 last 14.07) - open/close 18292 (time 1786818035.67 total 826.10 last 15.44) - open/close 18551 (time 1786818046.43 total 836.86 last 24.07) - open/close 18829 (time 1786818057.33 total 847.76 last 25.52) - open/close 19073 (time 1786818067.65 total 858.08 last 23.63) - open/close 19283 (time 1786818078.36 total 868.79 last 19.62) - open/close 19484 (time 1786818088.98 total 879.41 last 18.92) - open/close 20000 (time 1786818095.65 total 886.08 last 77.41) - open/close 20844 (time 1786818105.65 total 896.08 last 84.39) - open/close 21672 (time 1786818115.65 total 906.08 last 82.79) - open/close 22579 (time 1786818125.65 total 916.08 last 90.68) - open/close 23432 (time 1786818135.65 total 926.09 last 85.27) - open/close 24254 (time 1786818145.66 total 936.09 last 82.14) - open/close 24997 (time 1786818155.67 total 946.10 last 74.24) - open/close 25865 (time 1786818165.68 total 956.11 last 86.74) - open/close 26663 (time 1786818175.68 total 966.11 last 79.77) - open/close 27480 (time 1786818185.69 total 976.12 last 81.66) - open/close 28276 (time 1786818195.69 total 986.13 last 79.54) - open/close 29068 (time 1786818205.70 total 996.13 last 79.14) - open/close 29878 (time 1786818215.72 total 1006.15 last 80.87) - open/close 30733 (time 1786818225.72 total 1016.16 last 85.44) - open/close 31499 (time 1786818235.73 total 1026.16 last 76.56) - open/close 32285 (time 1786818245.73 total 1036.16 last 78.57) - open/close 33116 (time 1786818255.74 total 1046.18 last 83.01) - open/close 33853 (time 1786818265.75 total 1056.18 last 73.65) - open/close 34660 (time 1786818275.76 total 1066.19 last 80.62) - open/close 35511 (time 1786818285.77 total 1076.20 last 85.00) - open/close 36264 (time 1786818295.78 total 1086.21 last 75.25) - open/close 37093 (time 1786818305.78 total 1096.21 last 82.89) - open/close 37857 (time 1786818315.78 total 1106.21 last 76.39) - open/close 38630 (time 1786818325.79 total 1116.22 last 77.24) - open/close 39425 (time 1786818335.79 total 1126.22 last 79.48) - open/close 40000 (time 1786818342.90 total 1133.34 last 80.84) - open/close 40752 (time 1786818352.91 total 1143.34 last 75.19) - open/close 41535 (time 1786818362.91 total 1153.34 last 78.25) - open/close 42328 (time 1786818372.93 total 1163.36 last 79.18) - open/close 43173 (time 1786818382.94 total 1173.37 last 84.42) - open/close 43989 (time 1786818392.94 total 1183.37 last 81.57) - open/close 44770 (time 1786818402.95 total 1193.39 last 78.00) - open/close 45599 (time 1786818412.96 total 1203.40 last 82.82) - open/close 46443 (time 1786818422.97 total 1213.40 last 84.36) - open/close 47244 (time 1786818432.98 total 1223.41 last 80.03) - open/close 48073 (time 1786818442.98 total 1233.41 last 82.90) - open/close 48981 (time 1786818452.99 total 1243.42 last 90.71) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1251.02 seconds: 39.67 ops/second - unlinked 0 (time 1786818464 ; total 0 ; last 0) - unlinked 10000 (time 1786818600 ; total 136 ; last 136) - unlinked 20000 (time 1786818726 ; total 262 ; last 126) - unlinked 30000 (time 1786818857 ; total 393 ; last 131) - unlinked 40000 (time 1786818986 ; total 522 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 641 seconds: 77.427460 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg614-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg614-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg614-server stop mds service on oleg614-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg614-server stop mds service on oleg614-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg614-server unloading modules via unload_modules_local on: 'oleg614-server' oleg614-server: oleg614-server.virtnet: executing unload_modules_local oleg614-server: modules unloaded. pdsh@oleg614-client: oleg614-server: ssh exited with exit code 2 pdsh@oleg614-client: oleg614-client: ssh exited with exit code 2 pdsh@oleg614-client: oleg614-client: ssh exited with exit code 2 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 2