== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:30:38 (1710102638) start mds service on oleg245-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg245-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock oleg245-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 323 (time 1710102701.75 total 11.02 last 29.30) - open/close 498 (time 1710102711.76 total 21.03 last 17.48) - open/close 715 (time 1710102723.17 total 32.44 last 19.02) - open/close 967 (time 1710102733.17 total 42.45 last 25.19) - open/close 1167 (time 1710102743.47 total 52.74 last 19.43) - open/close 1336 (time 1710102753.47 total 62.74 last 16.90) - open/close 1590 (time 1710102763.48 total 72.75 last 25.37) - open/close 1777 (time 1710102773.48 total 82.75 last 18.70) - open/close 2079 (time 1710102783.60 total 92.87 last 29.84) - open/close 2334 (time 1710102793.60 total 102.88 last 25.49) - open/close 2592 (time 1710102803.63 total 112.90 last 25.74) - open/close 2860 (time 1710102814.62 total 123.89 last 24.39) - open/close 3119 (time 1710102825.43 total 134.70 last 23.95) - open/close 3362 (time 1710102836.21 total 145.48 last 22.55) - open/close 3614 (time 1710102846.93 total 156.21 last 23.49) - open/close 3858 (time 1710102857.80 total 167.07 last 22.47) - open/close 3973 (time 1710102870.05 total 179.32 last 9.38) - open/close 4138 (time 1710102880.80 total 190.07 last 15.35) - open/close 4329 (time 1710102890.96 total 200.23 last 18.80) - open/close 4585 (time 1710102900.96 total 210.23 last 25.59) - open/close 4689 (time 1710102911.16 total 220.43 last 10.20) - open/close 4811 (time 1710102921.56 total 230.83 last 11.73) - open/close 5088 (time 1710102932.30 total 241.58 last 25.79) - open/close 5147 (time 1710102942.96 total 252.23 last 5.54) - open/close 5402 (time 1710102953.39 total 262.66 last 24.43) - open/close 5720 (time 1710102964.14 total 273.41 last 29.60) - open/close 5985 (time 1710102974.43 total 283.70 last 25.76) - open/close 6300 (time 1710102984.75 total 294.02 last 30.52) - open/close 6463 (time 1710102994.87 total 304.14 last 16.10) - open/close 6762 (time 1710103005.07 total 314.34 last 29.32) - open/close 7005 (time 1710103015.84 total 325.11 last 22.55) - open/close 7253 (time 1710103025.85 total 335.12 last 24.78) - open/close 7629 (time 1710103035.85 total 345.12 last 37.60) - open/close 7965 (time 1710103046.33 total 355.60 last 32.06) - open/close 8688 (time 1710103056.34 total 365.61 last 72.28) - open/close 10000 (time 1710103065.83 total 375.10 last 138.26) - open/close 11302 (time 1710103075.83 total 385.10 last 130.20) - open/close 12538 (time 1710103085.83 total 395.10 last 123.56) - open/close 13761 (time 1710103095.83 total 405.10 last 122.28) - open/close 14956 (time 1710103105.84 total 415.11 last 119.42) - open/close 16218 (time 1710103115.84 total 425.12 last 126.11) - open/close 17375 (time 1710103125.85 total 435.12 last 115.62) - open/close 18491 (time 1710103135.86 total 445.13 last 111.53) - open/close 19733 (time 1710103145.86 total 455.13 last 124.19) - open/close 20000 (time 1710103148.06 total 457.33 last 121.11) - open/close 21322 (time 1710103158.07 total 467.34 last 132.15) - open/close 22605 (time 1710103168.07 total 477.34 last 128.28) - open/close 23801 (time 1710103178.07 total 487.34 last 119.57) - open/close 25027 (time 1710103188.08 total 497.35 last 122.54) - open/close 26173 (time 1710103198.08 total 507.35 last 114.57) - open/close 27261 (time 1710103208.09 total 517.36 last 108.69) - open/close 28420 (time 1710103218.09 total 527.37 last 115.84) - open/close 29396 (time 1710103228.10 total 537.37 last 97.54) - open/close 30000 (time 1710103234.34 total 543.61 last 96.85) - open/close 31156 (time 1710103244.34 total 553.61 last 115.55) - open/close 32135 (time 1710103254.34 total 563.61 last 97.89) - open/close 33129 (time 1710103264.34 total 573.61 last 99.39) - open/close 34110 (time 1710103274.35 total 583.62 last 98.05) - open/close 35086 (time 1710103284.35 total 593.62 last 97.57) - open/close 36013 (time 1710103294.35 total 603.63 last 92.68) - open/close 36893 (time 1710103304.36 total 613.63 last 87.93) - open/close 37741 (time 1710103314.37 total 623.64 last 84.77) - open/close 38571 (time 1710103324.37 total 633.64 last 82.98) - open/close 39361 (time 1710103334.37 total 643.64 last 79.00) - open/close 40000 (time 1710103342.03 total 651.30 last 83.45) - open/close 40863 (time 1710103352.03 total 661.30 last 86.28) - open/close 41689 (time 1710103362.04 total 671.31 last 82.51) - open/close 42499 (time 1710103372.05 total 681.32 last 80.91) - open/close 43224 (time 1710103382.05 total 691.32 last 72.50) - open/close 43902 (time 1710103392.06 total 701.34 last 67.70) - open/close 44590 (time 1710103402.07 total 711.34 last 68.76) - open/close 45235 (time 1710103412.08 total 721.35 last 64.43) - open/close 45892 (time 1710103422.09 total 731.36 last 65.63) - open/close 46525 (time 1710103432.10 total 741.37 last 63.25) - open/close 47123 (time 1710103442.11 total 751.38 last 59.75) - open/close 47722 (time 1710103452.12 total 761.40 last 59.80) - open/close 48408 (time 1710103462.12 total 771.40 last 68.60) - open/close 48998 (time 1710103472.14 total 781.41 last 58.90) - open/close 49547 (time 1710103482.14 total 791.41 last 54.89) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 792.96 seconds: 62.59 ops/second - unlinked 0 (time 1710103485 ; total 0 ; last 0) - unlinked 10000 (time 1710103616 ; total 131 ; last 131) - unlinked 20000 (time 1710103738 ; total 253 ; last 122) - unlinked 30000 (time 1710103825 ; total 340 ; last 87) - unlinked 40000 (time 1710103882 ; total 397 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 429 seconds: 115.692307 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server unloading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2