== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:12:45 (1746133965) start mds service on oleg437-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 149 (time 1746134005.37 total 10.05 last 14.82) - open/close 269 (time 1746134016.30 total 20.99 last 10.98) - open/close 471 (time 1746134026.30 total 30.99 last 20.20) - open/close 748 (time 1746134037.05 total 41.74 last 25.76) - open/close 1000 (time 1746134047.22 total 51.90 last 24.79) - open/close 1397 (time 1746134058.14 total 62.83 last 36.35) - open/close 1605 (time 1746134068.16 total 72.84 last 20.77) - open/close 2065 (time 1746134079.04 total 83.72 last 42.28) - open/close 2467 (time 1746134089.67 total 94.36 last 37.81) - open/close 2826 (time 1746134100.20 total 104.89 last 34.08) - open/close 3185 (time 1746134110.67 total 115.36 last 34.30) - open/close 3317 (time 1746134121.63 total 126.32 last 12.04) - open/close 3539 (time 1746134131.63 total 136.32 last 22.20) - open/close 3711 (time 1746134141.95 total 146.64 last 16.67) - open/close 3932 (time 1746134152.39 total 157.07 last 21.18) - open/close 4160 (time 1746134162.40 total 167.09 last 22.77) - open/close 4297 (time 1746134172.52 total 177.21 last 13.53) - open/close 4548 (time 1746134183.22 total 187.91 last 23.46) - open/close 4766 (time 1746134193.34 total 198.03 last 21.55) - open/close 5084 (time 1746134203.80 total 208.49 last 30.39) - open/close 5332 (time 1746134214.79 total 219.48 last 22.56) - open/close 5533 (time 1746134225.52 total 230.21 last 18.74) - open/close 5771 (time 1746134236.39 total 241.07 last 21.90) - open/close 6112 (time 1746134247.20 total 251.88 last 31.54) - open/close 6471 (time 1746134258.00 total 262.68 last 33.25) - open/close 6673 (time 1746134268.12 total 272.81 last 19.95) - open/close 6902 (time 1746134278.75 total 283.43 last 21.55) - open/close 7133 (time 1746134288.85 total 293.53 last 22.87) - open/close 7484 (time 1746134299.12 total 303.81 last 34.17) - open/close 7831 (time 1746134309.46 total 314.15 last 33.56) - open/close 8151 (time 1746134319.86 total 324.55 last 30.75) - open/close 8431 (time 1746134329.97 total 334.66 last 27.70) - open/close 10000 (time 1746134338.54 total 343.23 last 183.11) - open/close 12542 (time 1746134348.54 total 353.23 last 254.15) - open/close 14452 (time 1746134358.54 total 363.23 last 190.99) - open/close 16253 (time 1746134368.55 total 373.23 last 180.05) - open/close 17776 (time 1746134378.55 total 383.24 last 152.23) - open/close 19915 (time 1746134388.55 total 393.24 last 213.87) - open/close 22085 (time 1746134398.56 total 403.24 last 216.93) - open/close 23728 (time 1746134408.56 total 413.25 last 164.23) - open/close 25454 (time 1746134418.56 total 423.25 last 172.54) - open/close 27720 (time 1746134428.56 total 433.25 last 226.57) - open/close 29640 (time 1746134438.57 total 443.25 last 191.95) - open/close 31624 (time 1746134448.57 total 453.26 last 198.32) - open/close 33559 (time 1746134458.57 total 463.26 last 193.49) - open/close 35226 (time 1746134468.57 total 473.26 last 166.67) - open/close 36797 (time 1746134478.58 total 483.27 last 156.99) - open/close 38008 (time 1746134488.58 total 493.27 last 121.08) - open/close 39693 (time 1746134498.58 total 503.27 last 168.49) - open/close 41304 (time 1746134508.58 total 513.27 last 161.10) - open/close 42816 (time 1746134518.59 total 523.27 last 151.15) - open/close 44497 (time 1746134528.59 total 533.28 last 168.05) - open/close 46092 (time 1746134538.59 total 543.28 last 159.44) - open/close 47707 (time 1746134548.60 total 553.28 last 161.44) - open/close 49554 (time 1746134558.60 total 563.28 last 184.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 563.72 seconds: 88.04 ops/second - unlinked 0 (time 1746134560 ; total 0 ; last 0) - unlinked 10000 (time 1746134592 ; total 32 ; last 32) - unlinked 20000 (time 1746134627 ; total 67 ; last 35) - unlinked 30000 (time 1746134667 ; total 107 ; last 40) - unlinked 40000 (time 1746134695 ; total 135 ; last 28) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 168 seconds: 295.428558 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2