== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:39:58 (1788442798) start mds service on oleg201-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: kmemleak disabled oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg201-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg201-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg201-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 211 (time 1788442905.69 total 10.74 last 19.65) - open/close 481 (time 1788442915.70 total 20.75 last 26.97) - open/close 651 (time 1788442926.65 total 31.70 last 15.53) - open/close 925 (time 1788442936.67 total 41.71 last 27.35) - open/close 1176 (time 1788442947.68 total 52.72 last 22.80) - open/close 1371 (time 1788442957.70 total 62.75 last 19.45) - open/close 1549 (time 1788442968.02 total 73.07 last 17.25) - open/close 1728 (time 1788442978.03 total 83.08 last 17.88) - open/close 1884 (time 1788442988.04 total 93.08 last 15.59) - open/close 2090 (time 1788442998.04 total 103.09 last 20.59) - open/close 2345 (time 1788443008.34 total 113.38 last 24.77) - open/close 2501 (time 1788443019.67 total 124.71 last 13.77) - open/close 2615 (time 1788443029.69 total 134.74 last 11.37) - open/close 2735 (time 1788443039.90 total 144.94 last 11.76) - open/close 2859 (time 1788443050.96 total 156.00 last 11.21) - open/close 2967 (time 1788443061.39 total 166.43 last 10.36) - open/close 3211 (time 1788443071.40 total 176.44 last 24.37) - open/close 3394 (time 1788443081.84 total 186.89 last 17.52) - open/close 3506 (time 1788443092.76 total 197.81 last 10.26) - open/close 3756 (time 1788443102.80 total 207.84 last 24.92) - open/close 4026 (time 1788443112.80 total 217.84 last 26.99) - open/close 4158 (time 1788443122.80 total 227.85 last 13.19) - open/close 4277 (time 1788443132.81 total 237.85 last 11.90) - open/close 4431 (time 1788443142.83 total 247.87 last 15.37) - open/close 4612 (time 1788443152.85 total 257.90 last 18.05) - open/close 4693 (time 1788443163.09 total 268.13 last 7.91) - open/close 4911 (time 1788443173.42 total 278.46 last 21.11) - open/close 5076 (time 1788443183.42 total 288.47 last 16.50) - open/close 5246 (time 1788443193.90 total 298.95 last 16.22) - open/close 5373 (time 1788443204.50 total 309.54 last 11.99) - open/close 5507 (time 1788443214.82 total 319.86 last 12.98) - open/close 5726 (time 1788443224.84 total 329.89 last 21.85) - open/close 5918 (time 1788443234.92 total 339.97 last 19.05) - open/close 6165 (time 1788443245.02 total 350.07 last 24.45) - open/close 6204 (time 1788443255.33 total 360.37 last 3.79) - open/close 6346 (time 1788443266.29 total 371.34 last 12.95) - open/close 6565 (time 1788443276.30 total 381.34 last 21.88) - open/close 6831 (time 1788443286.30 total 391.35 last 26.60) - open/close 7038 (time 1788443296.47 total 401.52 last 20.36) - open/close 7088 (time 1788443307.25 total 412.29 last 4.64) - open/close 7229 (time 1788443319.70 total 424.74 last 11.32) - open/close 7476 (time 1788443329.70 total 434.75 last 24.70) - open/close 7584 (time 1788443340.46 total 445.50 last 10.04) - open/close 7704 (time 1788443350.47 total 455.51 last 11.99) - open/close 7959 (time 1788443360.47 total 465.51 last 25.49) - open/close 8200 (time 1788443371.36 total 476.41 last 22.13) - open/close 8405 (time 1788443381.36 total 486.41 last 20.50) - open/close 8477 (time 1788443392.61 total 497.65 last 6.40) - open/close 8606 (time 1788443402.98 total 508.02 last 12.44) - open/close 8849 (time 1788443412.99 total 518.04 last 24.27) - open/close 9049 (time 1788443423.00 total 528.05 last 19.98) - open/close 9108 (time 1788443433.82 total 538.87 last 5.45) - open/close 9247 (time 1788443443.99 total 549.04 last 13.67) - open/close 9268 (time 1788443454.94 total 559.99 last 1.92) - open/close 9446 (time 1788443465.76 total 570.81 last 16.45) - open/close 9701 (time 1788443478.70 total 583.75 last 19.71) - open/close 9934 (time 1788443489.59 total 594.63 last 21.40) - open/close 10000 (time 1788443493.29 total 598.34 last 17.81) - open/close 10196 (time 1788443503.31 total 608.36 last 19.56) - open/close 10310 (time 1788443513.57 total 618.62 last 11.11) - open/close 10468 (time 1788443524.57 total 629.62 last 14.36) - open/close 10668 (time 1788443534.58 total 639.62 last 19.99) - open/close 10912 (time 1788443547.43 total 652.47 last 18.99) - open/close 11089 (time 1788443558.11 total 663.15 last 16.57) - open/close 11231 (time 1788443568.12 total 673.16 last 14.19) - open/close 11340 (time 1788443578.42 total 683.47 last 10.58) - open/close 11479 (time 1788443588.63 total 693.68 last 13.61) - open/close 11690 (time 1788443598.65 total 703.70 last 21.07) - open/close 11931 (time 1788443609.37 total 714.41 last 22.49) - open/close 12197 (time 1788443620.19 total 725.24 last 24.57) - open/close 12347 (time 1788443630.61 total 735.66 last 14.40) - open/close 12484 (time 1788443640.65 total 745.70 last 13.64) - open/close 12730 (time 1788443650.66 total 755.70 last 24.59) - open/close 12955 (time 1788443661.27 total 766.31 last 21.21) - open/close 13235 (time 1788443671.27 total 776.32 last 27.99) - open/close 13477 (time 1788443681.29 total 786.34 last 24.15) - open/close 13612 (time 1788443691.30 total 796.35 last 13.49) - open/close 13877 (time 1788443701.30 total 806.35 last 26.50) - open/close 14077 (time 1788443711.32 total 816.37 last 19.97) - open/close 14199 (time 1788443722.24 total 827.29 last 11.17) - open/close 14283 (time 1788443732.27 total 837.31 last 8.38) - open/close 14439 (time 1788443742.27 total 847.32 last 15.60) - open/close 14654 (time 1788443752.30 total 857.34 last 21.44) - open/close 14762 (time 1788443762.73 total 867.78 last 10.35) - open/close 15010 (time 1788443772.74 total 877.78 last 24.79) - open/close 15243 (time 1788443783.38 total 888.43 last 21.89) - open/close 15391 (time 1788443795.28 total 900.33 last 12.43) - open/close 15670 (time 1788443806.18 total 911.23 last 25.60) - open/close 15927 (time 1788443816.53 total 921.58 last 24.83) - open/close 16071 (time 1788443826.84 total 931.88 last 13.97) - open/close 16190 (time 1788443837.67 total 942.72 last 10.99) - open/close 16314 (time 1788443847.76 total 952.81 last 12.29) - open/close 16459 (time 1788443858.66 total 963.70 last 13.31) - open/close 16580 (time 1788443869.60 total 974.65 last 11.06) - open/close 16707 (time 1788443879.77 total 984.82 last 12.49) - open/close 16837 (time 1788443889.97 total 995.01 last 12.75) - open/close 17021 (time 1788443899.97 total 1005.02 last 18.39) - open/close 17238 (time 1788443909.98 total 1015.03 last 21.68) - open/close 17360 (time 1788443920.29 total 1025.34 last 11.83) - open/close 17508 (time 1788443930.98 total 1036.02 last 13.85) - open/close 17630 (time 1788443941.78 total 1046.83 last 11.29) - open/close 17729 (time 1788443951.79 total 1056.84 last 9.89) - open/close 17822 (time 1788443962.61 total 1067.66 last 8.60) - open/close 18007 (time 1788443972.71 total 1077.76 last 18.31) - open/close 18649 (time 1788443982.72 total 1087.77 last 64.15) - open/close 19450 (time 1788443992.72 total 1097.77 last 80.10) - open/close 20000 (time 1788444000.16 total 1105.21 last 73.92) - open/close 20784 (time 1788444010.17 total 1115.21 last 78.38) - open/close 21561 (time 1788444020.17 total 1125.21 last 77.69) - open/close 22341 (time 1788444030.17 total 1135.22 last 77.95) - open/close 23126 (time 1788444040.18 total 1145.23 last 78.44) - open/close 23852 (time 1788444050.19 total 1155.23 last 72.55) - open/close 24484 (time 1788444060.19 total 1165.24 last 63.19) - open/close 25051 (time 1788444070.21 total 1175.25 last 56.61) - open/close 25722 (time 1788444080.22 total 1185.27 last 67.00) - open/close 26443 (time 1788444090.23 total 1195.27 last 72.06) - open/close 26978 (time 1788444100.23 total 1205.28 last 53.45) - open/close 27635 (time 1788444110.24 total 1215.28 last 65.70) - open/close 28228 (time 1788444120.24 total 1225.29 last 59.25) - open/close 28878 (time 1788444130.25 total 1235.30 last 64.95) - open/close 29703 (time 1788444140.26 total 1245.30 last 82.48) - open/close 30000 (time 1788444143.95 total 1249.00 last 80.39) - open/close 30846 (time 1788444153.96 total 1259.00 last 84.52) - open/close 31650 (time 1788444163.97 total 1269.01 last 80.33) - open/close 32232 (time 1788444173.97 total 1279.02 last 58.17) - open/close 32999 (time 1788444183.99 total 1289.03 last 76.58) - open/close 33832 (time 1788444194.00 total 1299.04 last 83.21) - open/close 34577 (time 1788444204.01 total 1309.05 last 74.43) - open/close 35315 (time 1788444214.01 total 1319.06 last 73.77) - open/close 36083 (time 1788444224.03 total 1329.07 last 76.67) - open/close 36853 (time 1788444234.04 total 1339.08 last 76.94) - open/close 37493 (time 1788444244.04 total 1349.09 last 63.94) - open/close 38349 (time 1788444254.05 total 1359.09 last 85.60) - open/close 39219 (time 1788444264.06 total 1369.11 last 86.87) - open/close 40000 (time 1788444273.91 total 1378.95 last 79.31) - open/close 40644 (time 1788444283.91 total 1388.96 last 64.36) - open/close 41369 (time 1788444293.95 total 1398.99 last 72.25) - open/close 42213 (time 1788444303.95 total 1409.00 last 84.36) - open/close 43038 (time 1788444313.96 total 1419.00 last 82.46) - open/close 43817 (time 1788444323.99 total 1429.04 last 77.63) - open/close 44569 (time 1788444334.00 total 1439.04 last 75.16) - open/close 45248 (time 1788444344.01 total 1449.06 last 67.82) - open/close 45934 (time 1788444354.01 total 1459.06 last 68.59) - open/close 46673 (time 1788444364.01 total 1469.06 last 73.88) - open/close 47220 (time 1788444374.02 total 1479.06 last 54.69) - open/close 47938 (time 1788444384.02 total 1489.07 last 71.76) - open/close 48768 (time 1788444394.03 total 1499.07 last 82.96) - open/close 49571 (time 1788444404.03 total 1509.08 last 80.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1509.91 seconds: 32.87 ops/second - unlinked 0 (time 1788444409 ; total 0 ; last 0) - unlinked 10000 (time 1788444536 ; total 127 ; last 127) - unlinked 20000 (time 1788444661 ; total 252 ; last 125) - unlinked 30000 (time 1788444797 ; total 388 ; last 136) - unlinked 40000 (time 1788444936 ; total 527 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 664 seconds: 74.745483 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg201-server unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: kmemleak disabled modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2