== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:19:13 (1718853553) start mds service on oleg233-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: detected 4 online CPUs by sysfs oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg233-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg233-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock oleg233-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1718853594.76 total 10.98 last 22.31) - open/close 918 (time 1718853605.45 total 21.67 last 62.99) - open/close 1560 (time 1718853615.74 total 31.96 last 62.40) - open/close 1868 (time 1718853629.76 total 45.98 last 21.97) - open/close 2190 (time 1718853640.50 total 56.72 last 29.98) - open/close 2679 (time 1718853650.57 total 66.79 last 48.58) - open/close 3027 (time 1718853661.46 total 77.68 last 31.93) - open/close 3573 (time 1718853672.36 total 88.58 last 50.13) - open/close 4073 (time 1718853682.42 total 98.64 last 49.66) - open/close 4773 (time 1718853692.71 total 108.93 last 68.05) - open/close 5205 (time 1718853702.71 total 118.93 last 43.20) - open/close 5684 (time 1718853714.12 total 130.34 last 41.99) - open/close 6032 (time 1718853724.22 total 140.45 last 34.44) - open/close 6555 (time 1718853735.16 total 151.38 last 47.81) - open/close 7116 (time 1718853745.40 total 161.62 last 54.79) - open/close 7586 (time 1718853755.54 total 171.76 last 46.36) - open/close 7876 (time 1718853766.38 total 182.60 last 26.76) - open/close 8485 (time 1718853776.58 total 192.80 last 59.70) - open/close 9124 (time 1718853786.82 total 203.04 last 62.39) - open/close 9703 (time 1718853797.12 total 213.34 last 56.24) - open/close 10000 (time 1718853802.71 total 218.93 last 53.09) - open/close 10540 (time 1718853812.82 total 229.04 last 53.43) - open/close 11085 (time 1718853823.06 total 239.28 last 53.21) - open/close 11765 (time 1718853833.80 total 250.02 last 63.32) - open/close 12248 (time 1718853844.30 total 260.52 last 45.99) - open/close 16316 (time 1718853854.30 total 270.52 last 406.77) - open/close 20000 (time 1718853863.20 total 279.42 last 413.97) - open/close 24462 (time 1718853873.20 total 289.42 last 446.15) - open/close 29347 (time 1718853883.21 total 299.43 last 488.41) - open/close 34435 (time 1718853893.21 total 309.43 last 508.71) - open/close 39124 (time 1718853903.21 total 319.43 last 468.82) - open/close 40000 (time 1718853905.24 total 321.47 last 430.22) - open/close 44472 (time 1718853915.25 total 331.47 last 447.15) - open/close 48525 (time 1718853925.25 total 341.47 last 405.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 344.86 seconds: 143.92 ops/second - unlinked 0 (time 1718853929 ; total 0 ; last 0) - unlinked 10000 (time 1718853941 ; total 12 ; last 12) - unlinked 20000 (time 1718853952 ; total 23 ; last 11) - unlinked 30000 (time 1718853964 ; total 35 ; last 12) - unlinked 40000 (time 1718853977 ; total 48 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 59 seconds: 841.220337 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server unloading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2