== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:33:47 (1788428027) start mds service on oleg218-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921464 Sep 3 03:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: kmemleak disabled oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg218-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg218-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 237 (time 1788428155.87 total 10.01 last 23.67) - open/close 341 (time 1788428165.88 total 20.02 last 10.39) - open/close 445 (time 1788428176.94 total 31.09 last 9.40) - open/close 549 (time 1788428187.83 total 41.97 last 9.55) - open/close 679 (time 1788428198.11 total 52.26 last 12.64) - open/close 797 (time 1788428208.48 total 62.63 last 11.37) - open/close 950 (time 1788428219.48 total 73.63 last 13.91) - open/close 1143 (time 1788428229.49 total 83.64 last 19.28) - open/close 1276 (time 1788428239.51 total 93.66 last 13.28) - open/close 1546 (time 1788428249.53 total 103.67 last 26.95) - open/close 1734 (time 1788428259.75 total 113.90 last 18.38) - open/close 1857 (time 1788428270.06 total 124.20 last 11.94) - open/close 2126 (time 1788428280.37 total 134.52 last 26.08) - open/close 2397 (time 1788428290.47 total 144.62 last 26.84) - open/close 2600 (time 1788428301.52 total 155.66 last 18.38) - open/close 2747 (time 1788428311.87 total 166.01 last 14.20) - open/close 2940 (time 1788428322.60 total 176.74 last 17.98) - open/close 3120 (time 1788428333.60 total 187.74 last 16.36) - open/close 3277 (time 1788428344.23 total 198.37 last 14.77) - open/close 3399 (time 1788428354.27 total 208.42 last 12.15) - open/close 3666 (time 1788428364.28 total 218.43 last 26.66) - open/close 3890 (time 1788428374.36 total 228.51 last 22.23) - open/close 4068 (time 1788428384.38 total 238.53 last 17.77) - open/close 4136 (time 1788428394.65 total 248.80 last 6.62) - open/close 4268 (time 1788428404.66 total 258.81 last 13.18) - open/close 4435 (time 1788428414.67 total 268.82 last 16.69) - open/close 4616 (time 1788428424.68 total 278.83 last 18.08) - open/close 4727 (time 1788428435.43 total 289.58 last 10.32) - open/close 4886 (time 1788428445.91 total 300.06 last 15.17) - open/close 5107 (time 1788428456.93 total 311.07 last 20.07) - open/close 5211 (time 1788428467.83 total 321.98 last 9.54) - open/close 5287 (time 1788428477.87 total 332.02 last 7.57) - open/close 5518 (time 1788428487.88 total 342.03 last 23.07) - open/close 5673 (time 1788428498.50 total 352.64 last 14.60) - open/close 5836 (time 1788428508.50 total 362.65 last 16.30) - open/close 5970 (time 1788428518.52 total 372.67 last 13.37) - open/close 6168 (time 1788428528.55 total 382.69 last 19.75) - open/close 6275 (time 1788428541.23 total 395.37 last 8.44) - open/close 6593 (time 1788428551.40 total 405.55 last 31.24) - open/close 6792 (time 1788428562.02 total 416.17 last 18.74) - open/close 6868 (time 1788428572.26 total 426.41 last 7.42) - open/close 7007 (time 1788428582.28 total 436.43 last 13.87) - open/close 7123 (time 1788428592.49 total 446.63 last 11.37) - open/close 7149 (time 1788428603.53 total 457.68 last 2.35) - open/close 7344 (time 1788428613.54 total 467.69 last 19.48) - open/close 7536 (time 1788428623.55 total 477.69 last 19.19) - open/close 7731 (time 1788428635.30 total 489.45 last 16.59) - open/close 7833 (time 1788428645.87 total 500.01 last 9.66) - open/close 8079 (time 1788428656.69 total 510.84 last 22.72) - open/close 8354 (time 1788428666.70 total 520.84 last 27.48) - open/close 8594 (time 1788428677.22 total 531.37 last 22.80) - open/close 8830 (time 1788428688.41 total 542.56 last 21.10) - open/close 9145 (time 1788428699.31 total 553.46 last 28.89) - open/close 9341 (time 1788428709.74 total 563.89 last 18.80) - open/close 9484 (time 1788428722.55 total 576.70 last 11.16) - open/close 9705 (time 1788428732.56 total 586.70 last 22.10) - open/close 9809 (time 1788428743.19 total 597.34 last 9.78) - open/close 9933 (time 1788428753.83 total 607.98 last 11.66) - open/close 10000 (time 1788428758.15 total 612.30 last 15.51) - open/close 10171 (time 1788428768.77 total 622.92 last 16.10) - open/close 10476 (time 1788428778.78 total 632.92 last 30.49) - open/close 10722 (time 1788428788.78 total 642.93 last 24.59) - open/close 10915 (time 1788428799.65 total 653.79 last 17.76) - open/close 11070 (time 1788428809.72 total 663.86 last 15.40) - open/close 11276 (time 1788428819.73 total 673.88 last 20.56) - open/close 11434 (time 1788428830.11 total 684.26 last 15.22) - open/close 11628 (time 1788428840.19 total 694.34 last 19.24) - open/close 11838 (time 1788428850.80 total 704.95 last 19.80) - open/close 12104 (time 1788428860.84 total 714.99 last 26.49) - open/close 12390 (time 1788428870.86 total 725.00 last 28.56) - open/close 12528 (time 1788428881.47 total 735.62 last 13.00) - open/close 12728 (time 1788428892.27 total 746.41 last 18.52) - open/close 12981 (time 1788428902.38 total 756.53 last 25.02) - open/close 13195 (time 1788428913.34 total 767.49 last 19.52) - open/close 13420 (time 1788428923.85 total 778.00 last 21.41) - open/close 13702 (time 1788428935.79 total 789.93 last 23.63) - open/close 13849 (time 1788428947.80 total 801.94 last 12.24) - open/close 14030 (time 1788428958.63 total 812.78 last 16.71) - open/close 14284 (time 1788428969.47 total 823.61 last 23.44) - open/close 14450 (time 1788428980.06 total 834.21 last 15.67) - open/close 14718 (time 1788428990.07 total 844.22 last 26.77) - open/close 14894 (time 1788429001.06 total 855.21 last 16.01) - open/close 14990 (time 1788429012.02 total 866.17 last 8.76) - open/close 15163 (time 1788429022.02 total 876.17 last 17.30) - open/close 15357 (time 1788429032.58 total 886.73 last 18.37) - open/close 15600 (time 1788429042.59 total 896.74 last 24.27) - open/close 15722 (time 1788429054.94 total 909.09 last 9.88) - open/close 15864 (time 1788429065.46 total 919.61 last 13.50) - open/close 16072 (time 1788429075.75 total 929.89 last 20.22) - open/close 16148 (time 1788429085.94 total 940.09 last 7.46) - open/close 16298 (time 1788429096.55 total 950.69 last 14.14) - open/close 16428 (time 1788429106.85 total 961.00 last 12.61) - open/close 16606 (time 1788429116.90 total 971.04 last 17.72) - open/close 16781 (time 1788429126.93 total 981.08 last 17.45) - open/close 16962 (time 1788429136.94 total 991.09 last 18.07) - open/close 17187 (time 1788429147.83 total 1001.97 last 20.67) - open/close 17345 (time 1788429160.35 total 1014.49 last 12.62) - open/close 17512 (time 1788429171.42 total 1025.57 last 15.08) - open/close 17640 (time 1788429181.53 total 1035.67 last 12.67) - open/close 17796 (time 1788429191.65 total 1045.80 last 15.41) - open/close 17980 (time 1788429201.68 total 1055.82 last 18.35) - open/close 18109 (time 1788429211.93 total 1066.08 last 12.58) - open/close 18292 (time 1788429221.93 total 1076.08 last 18.29) - open/close 18486 (time 1788429231.95 total 1086.10 last 19.36) - open/close 19229 (time 1788429241.96 total 1096.10 last 74.27) - open/close 19975 (time 1788429251.96 total 1106.11 last 74.59) - open/close 20716 (time 1788429261.97 total 1116.12 last 74.02) - open/close 21491 (time 1788429271.98 total 1126.13 last 77.42) - open/close 22368 (time 1788429281.98 total 1136.13 last 87.65) - open/close 23045 (time 1788429292.00 total 1146.14 last 67.62) - open/close 23773 (time 1788429302.00 total 1156.15 last 72.78) - open/close 24463 (time 1788429312.00 total 1166.15 last 68.99) - open/close 25041 (time 1788429322.02 total 1176.17 last 57.69) - open/close 25894 (time 1788429332.02 total 1186.17 last 85.30) - open/close 26736 (time 1788429342.04 total 1196.19 last 84.03) - open/close 27474 (time 1788429352.05 total 1206.20 last 73.73) - open/close 28286 (time 1788429362.05 total 1216.20 last 81.18) - open/close 29060 (time 1788429372.06 total 1226.20 last 77.36) - open/close 29828 (time 1788429382.06 total 1236.21 last 76.78) - open/close 30000 (time 1788429384.58 total 1238.73 last 68.15) - open/close 30737 (time 1788429394.60 total 1248.75 last 73.58) - open/close 31498 (time 1788429404.60 total 1258.75 last 76.07) - open/close 32397 (time 1788429414.61 total 1268.75 last 89.86) - open/close 33196 (time 1788429424.62 total 1278.77 last 79.79) - open/close 33954 (time 1788429434.63 total 1288.78 last 75.72) - open/close 34751 (time 1788429444.63 total 1298.78 last 79.68) - open/close 35539 (time 1788429454.64 total 1308.78 last 78.78) - open/close 36347 (time 1788429464.66 total 1318.80 last 80.66) - open/close 37087 (time 1788429474.67 total 1328.82 last 73.88) - open/close 37751 (time 1788429484.68 total 1338.83 last 66.34) - open/close 38332 (time 1788429494.69 total 1348.83 last 58.07) - open/close 39122 (time 1788429504.69 total 1358.84 last 78.98) - open/close 39939 (time 1788429514.69 total 1368.84 last 81.65) - open/close 40701 (time 1788429524.70 total 1378.85 last 76.14) - open/close 41392 (time 1788429534.71 total 1388.86 last 69.02) - open/close 42284 (time 1788429544.72 total 1398.87 last 89.15) - open/close 43199 (time 1788429554.73 total 1408.87 last 91.45) - open/close 43936 (time 1788429564.73 total 1418.88 last 73.65) - open/close 44670 (time 1788429574.74 total 1428.89 last 73.33) - open/close 45432 (time 1788429584.75 total 1438.90 last 76.13) - open/close 46177 (time 1788429594.76 total 1448.90 last 74.47) - open/close 46881 (time 1788429604.77 total 1458.91 last 70.33) - open/close 47643 (time 1788429614.77 total 1468.92 last 76.15) - open/close 48399 (time 1788429624.78 total 1478.93 last 75.53) - open/close 49158 (time 1788429634.79 total 1488.94 last 75.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1494.37 seconds: 33.21 ops/second - unlinked 0 (time 1788429644 ; total 0 ; last 0) - unlinked 10000 (time 1788429788 ; total 144 ; last 144) - unlinked 20000 (time 1788429918 ; total 274 ; last 130) - unlinked 30000 (time 1788430049 ; total 405 ; last 131) - unlinked 40000 (time 1788430182 ; total 538 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 673 seconds: 73.745911 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: kmemleak disabled modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2