== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:57:38 (1746154658) start mds service on oleg255-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1746154714.64 total 10.01 last 18.57) - open/close 380 (time 1746154724.76 total 20.14 last 19.16) - open/close 579 (time 1746154735.12 total 30.49 last 19.22) - open/close 728 (time 1746154745.97 total 41.35 last 13.72) - open/close 959 (time 1746154756.39 total 51.77 last 22.17) - open/close 1117 (time 1746154767.08 total 62.46 last 14.79) - open/close 1208 (time 1746154777.23 total 72.61 last 8.97) - open/close 1301 (time 1746154787.64 total 83.02 last 8.93) - open/close 1435 (time 1746154798.05 total 93.42 last 12.87) - open/close 1727 (time 1746154808.26 total 103.64 last 28.59) - open/close 1853 (time 1746154818.65 total 114.02 last 12.13) - open/close 2148 (time 1746154829.36 total 124.74 last 27.54) - open/close 2317 (time 1746154839.64 total 135.02 last 16.44) - open/close 2491 (time 1746154850.29 total 145.67 last 16.33) - open/close 2735 (time 1746154860.60 total 155.97 last 23.68) - open/close 2844 (time 1746154871.80 total 167.18 last 9.73) - open/close 3007 (time 1746154881.80 total 177.18 last 16.30) - open/close 3222 (time 1746154891.92 total 187.30 last 21.24) - open/close 3492 (time 1746154902.32 total 197.69 last 25.97) - open/close 3690 (time 1746154912.91 total 208.29 last 18.69) - open/close 3899 (time 1746154922.91 total 218.29 last 20.89) - open/close 4119 (time 1746154932.92 total 228.29 last 21.99) - open/close 4309 (time 1746154943.02 total 238.40 last 18.80) - open/close 4641 (time 1746154953.93 total 249.31 last 30.44) - open/close 4880 (time 1746154964.39 total 259.77 last 22.85) - open/close 5165 (time 1746154974.39 total 269.77 last 28.49) - open/close 5447 (time 1746154984.74 total 280.12 last 27.25) - open/close 5710 (time 1746154994.88 total 290.26 last 25.94) - open/close 5939 (time 1746155005.69 total 301.06 last 21.19) - open/close 6190 (time 1746155016.31 total 311.69 last 23.62) - open/close 6426 (time 1746155026.70 total 322.08 last 22.72) - open/close 6701 (time 1746155037.38 total 332.76 last 25.75) - open/close 6945 (time 1746155048.61 total 343.98 last 21.74) - open/close 7116 (time 1746155059.61 total 354.99 last 15.54) - open/close 7332 (time 1746155070.20 total 365.57 last 20.41) - open/close 7669 (time 1746155081.21 total 376.59 last 30.59) - open/close 7937 (time 1746155091.42 total 386.80 last 26.24) - open/close 8127 (time 1746155102.77 total 398.15 last 16.75) - open/close 8295 (time 1746155112.77 total 408.15 last 16.80) - open/close 8613 (time 1746155122.97 total 418.35 last 31.17) - open/close 8876 (time 1746155133.78 total 429.16 last 24.33) - open/close 9097 (time 1746155144.42 total 439.79 last 20.78) - open/close 9233 (time 1746155154.42 total 449.80 last 13.60) - open/close 9399 (time 1746155164.87 total 460.25 last 15.89) - open/close 9708 (time 1746155175.88 total 471.26 last 28.06) - open/close 10000 (time 1746155183.83 total 479.21 last 36.74) - open/close 11453 (time 1746155193.83 total 489.21 last 145.28) - open/close 12823 (time 1746155203.83 total 499.21 last 137.00) - open/close 14272 (time 1746155213.83 total 509.21 last 144.88) - open/close 15705 (time 1746155223.83 total 519.21 last 143.30) - open/close 17107 (time 1746155233.84 total 529.21 last 140.16) - open/close 18331 (time 1746155243.84 total 539.21 last 122.40) - open/close 19554 (time 1746155253.84 total 549.22 last 122.28) - open/close 20000 (time 1746155257.06 total 552.44 last 138.44) - open/close 21295 (time 1746155267.06 total 562.44 last 129.49) - open/close 22697 (time 1746155277.06 total 572.44 last 140.17) - open/close 23974 (time 1746155287.06 total 582.44 last 127.70) - open/close 25257 (time 1746155297.07 total 592.44 last 128.25) - open/close 26569 (time 1746155307.07 total 602.45 last 131.15) - open/close 27867 (time 1746155317.07 total 612.45 last 129.78) - open/close 29237 (time 1746155327.07 total 622.45 last 136.98) - open/close 30000 (time 1746155332.48 total 627.86 last 141.15) - open/close 31438 (time 1746155342.49 total 637.87 last 143.66) - open/close 33041 (time 1746155352.49 total 647.87 last 160.26) - open/close 34566 (time 1746155362.49 total 657.87 last 152.48) - open/close 35975 (time 1746155372.50 total 667.88 last 140.81) - open/close 37353 (time 1746155382.50 total 677.88 last 137.76) - open/close 38661 (time 1746155392.51 total 687.88 last 130.76) - open/close 39930 (time 1746155402.51 total 697.88 last 126.87) - open/close 41400 (time 1746155412.51 total 707.89 last 146.97) - open/close 42649 (time 1746155422.51 total 717.89 last 124.89) - open/close 43971 (time 1746155432.51 total 727.89 last 132.18) - open/close 45208 (time 1746155442.53 total 737.90 last 123.48) - open/close 46557 (time 1746155452.53 total 747.91 last 134.83) - open/close 47845 (time 1746155462.54 total 757.92 last 128.67) - open/close 49034 (time 1746155472.55 total 767.93 last 118.84) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 772.74 seconds: 64.23 ops/second - unlinked 0 (time 1746155479 ; total 0 ; last 0) - unlinked 10000 (time 1746155536 ; total 57 ; last 57) - unlinked 20000 (time 1746155586 ; total 107 ; last 50) - unlinked 30000 (time 1746155627 ; total 148 ; last 41) - unlinked 40000 (time 1746155666 ; total 187 ; last 39) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 224 seconds: 221.571426 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2