== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:38:08 (1718318288) start mds service on oleg249-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg249-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock oleg249-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 390 (time 1718318328.20 total 10.33 last 37.75) - open/close 710 (time 1718318338.27 total 20.39 last 31.80) - open/close 1092 (time 1718318349.03 total 31.15 last 35.50) - open/close 1888 (time 1718318359.48 total 41.60 last 76.19) - open/close 2589 (time 1718318369.75 total 51.87 last 68.25) - open/close 3188 (time 1718318379.89 total 62.02 last 59.06) - open/close 3719 (time 1718318390.01 total 72.14 last 52.46) - open/close 4157 (time 1718318400.29 total 82.42 last 42.61) - open/close 4560 (time 1718318410.36 total 92.49 last 40.02) - open/close 5014 (time 1718318421.03 total 103.16 last 42.54) - open/close 5535 (time 1718318431.78 total 113.91 last 48.46) - open/close 6035 (time 1718318442.20 total 124.33 last 47.99) - open/close 6475 (time 1718318452.62 total 134.74 last 42.26) - open/close 6935 (time 1718318462.96 total 145.09 last 44.47) - open/close 7578 (time 1718318473.41 total 155.54 last 61.52) - open/close 7962 (time 1718318483.49 total 165.61 last 38.12) - open/close 8460 (time 1718318493.96 total 176.08 last 47.57) - open/close 8754 (time 1718318503.96 total 186.08 last 29.40) - open/close 9046 (time 1718318514.92 total 197.05 last 26.63) - open/close 9195 (time 1718318527.51 total 209.64 last 11.83) - open/close 9578 (time 1718318537.70 total 219.83 last 37.60) - open/close 9961 (time 1718318548.01 total 230.14 last 37.15) - open/close 10000 (time 1718318552.18 total 234.31 last 9.35) - open/close 10550 (time 1718318564.82 total 246.95 last 43.52) - open/close 10891 (time 1718318574.93 total 257.06 last 33.72) - open/close 11245 (time 1718318585.17 total 267.30 last 34.57) - open/close 11527 (time 1718318596.03 total 278.15 last 25.98) - open/close 12003 (time 1718318606.67 total 288.79 last 44.73) - open/close 13645 (time 1718318616.67 total 298.80 last 164.15) - open/close 17312 (time 1718318626.67 total 308.80 last 366.65) - open/close 20000 (time 1718318633.66 total 315.78 last 384.88) - open/close 24325 (time 1718318643.66 total 325.78 last 432.48) - open/close 29130 (time 1718318653.66 total 335.79 last 480.40) - open/close 33915 (time 1718318663.66 total 345.79 last 478.45) - open/close 37234 (time 1718318673.66 total 355.79 last 331.79) - open/close 40000 (time 1718318681.77 total 363.90 last 340.97) - open/close 44012 (time 1718318691.78 total 373.90 last 401.09) - open/close 47288 (time 1718318701.78 total 383.91 last 327.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 390.41 seconds: 127.13 ops/second - unlinked 0 (time 1718318709 ; total 0 ; last 0) - unlinked 10000 (time 1718318724 ; total 15 ; last 15) - unlinked 20000 (time 1718318741 ; total 32 ; last 17) - unlinked 30000 (time 1718318759 ; total 50 ; last 18) - unlinked 40000 (time 1718318779 ; total 70 ; last 20) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 82 seconds: 605.268311 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server unloading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2