== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:42:30 (1722796950) start mds service on oleg424-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg424-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock oleg424-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 525 (time 1722796989.66 total 10.18 last 51.56) - open/close 1043 (time 1722796999.77 total 20.29 last 51.24) - open/close 1539 (time 1722797009.84 total 30.36 last 49.25) - open/close 2075 (time 1722797020.00 total 40.52 last 52.75) - open/close 2575 (time 1722797030.00 total 50.52 last 50.00) - open/close 3062 (time 1722797040.00 total 60.53 last 48.69) - open/close 3543 (time 1722797050.00 total 70.53 last 48.09) - open/close 4023 (time 1722797060.02 total 80.54 last 47.94) - open/close 4505 (time 1722797070.04 total 90.57 last 48.07) - open/close 4994 (time 1722797080.10 total 100.63 last 48.60) - open/close 5484 (time 1722797090.17 total 110.70 last 48.66) - open/close 5986 (time 1722797100.30 total 120.82 last 49.59) - open/close 6484 (time 1722797110.41 total 130.94 last 49.22) - open/close 6967 (time 1722797120.44 total 140.96 last 48.17) - open/close 7452 (time 1722797130.49 total 151.01 last 48.26) - open/close 7943 (time 1722797140.56 total 161.09 last 48.75) - open/close 8430 (time 1722797150.62 total 171.15 last 48.41) - open/close 8925 (time 1722797160.73 total 181.25 last 48.99) - open/close 9426 (time 1722797170.80 total 191.33 last 49.71) - open/close 9931 (time 1722797180.87 total 201.39 last 50.16) - open/close 10455 (time 1722797191.01 total 211.53 last 51.67) - open/close 11024 (time 1722797201.34 total 221.86 last 55.09) - open/close 11585 (time 1722797211.66 total 232.18 last 54.37) - open/close 12136 (time 1722797221.93 total 242.46 last 53.61) - open/close 12647 (time 1722797232.04 total 252.56 last 50.59) - open/close 13151 (time 1722797242.12 total 262.65 last 49.96) - open/close 13663 (time 1722797252.25 total 272.78 last 50.55) - open/close 14178 (time 1722797262.39 total 282.91 last 50.80) - open/close 14706 (time 1722797272.53 total 293.06 last 52.06) - open/close 15219 (time 1722797282.65 total 303.17 last 50.71) - open/close 15734 (time 1722797292.73 total 313.25 last 51.11) - open/close 17653 (time 1722797302.73 total 323.25 last 191.90) - open/close 20000 (time 1722797312.43 total 332.95 last 241.91) - open/close 22425 (time 1722797322.43 total 342.95 last 242.41) - open/close 24863 (time 1722797332.43 total 352.96 last 243.74) - open/close 27262 (time 1722797342.43 total 362.96 last 239.88) - open/close 29606 (time 1722797352.44 total 372.96 last 234.35) - open/close 31925 (time 1722797362.44 total 382.96 last 231.89) - open/close 34241 (time 1722797372.44 total 392.96 last 231.55) - open/close 36579 (time 1722797382.44 total 402.97 last 233.71) - open/close 38898 (time 1722797392.45 total 412.97 last 231.83) - open/close 40000 (time 1722797397.27 total 417.79 last 228.48) - open/close 42292 (time 1722797407.27 total 427.79 last 229.17) - open/close 44619 (time 1722797417.27 total 437.80 last 232.60) - open/close 46933 (time 1722797427.28 total 447.80 last 231.35) - open/close 49205 (time 1722797437.28 total 457.80 last 227.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 459.65 seconds: 107.98 ops/second - unlinked 0 (time 1722797440 ; total 0 ; last 0) - unlinked 10000 (time 1722797464 ; total 24 ; last 24) - unlinked 20000 (time 1722797487 ; total 47 ; last 23) - unlinked 30000 (time 1722797510 ; total 70 ; last 23) - unlinked 40000 (time 1722797534 ; total 94 ; last 24) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 116 seconds: 427.862061 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server unloading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2