== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:51:07 (1734677467) start mds service on oleg339-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg339-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1734677519.72 total 10.06 last 25.55) - open/close 495 (time 1734677530.19 total 20.53 last 22.73) - open/close 799 (time 1734677540.60 total 30.94 last 29.20) - open/close 968 (time 1734677550.78 total 41.12 last 16.60) - open/close 1217 (time 1734677561.36 total 51.70 last 23.54) - open/close 1376 (time 1734677572.37 total 62.71 last 14.44) - open/close 1529 (time 1734677582.42 total 72.76 last 15.22) - open/close 1715 (time 1734677592.59 total 82.94 last 18.29) - open/close 2029 (time 1734677603.50 total 93.85 last 28.78) - open/close 2296 (time 1734677613.79 total 104.13 last 25.97) - open/close 2577 (time 1734677624.24 total 114.58 last 26.89) - open/close 2771 (time 1734677634.25 total 124.59 last 19.38) - open/close 2968 (time 1734677645.02 total 135.36 last 18.28) - open/close 3156 (time 1734677655.61 total 145.95 last 17.76) - open/close 3416 (time 1734677666.13 total 156.47 last 24.71) - open/close 3628 (time 1734677676.14 total 166.48 last 21.18) - open/close 3798 (time 1734677686.83 total 177.18 last 15.90) - open/close 4042 (time 1734677697.60 total 187.95 last 22.66) - open/close 4238 (time 1734677707.88 total 198.22 last 19.08) - open/close 4429 (time 1734677718.11 total 208.45 last 18.68) - open/close 4693 (time 1734677728.77 total 219.12 last 24.75) - open/close 4971 (time 1734677739.18 total 229.52 last 26.71) - open/close 5352 (time 1734677749.18 total 239.52 last 38.09) - open/close 5722 (time 1734677759.98 total 250.32 last 34.27) - open/close 6068 (time 1734677770.71 total 261.05 last 32.24) - open/close 6294 (time 1734677781.21 total 271.56 last 21.52) - open/close 6620 (time 1734677791.56 total 281.90 last 31.51) - open/close 6894 (time 1734677801.58 total 291.93 last 27.33) - open/close 7143 (time 1734677811.83 total 302.17 last 24.30) - open/close 7443 (time 1734677822.12 total 312.47 last 29.14) - open/close 7661 (time 1734677832.37 total 322.71 last 21.29) - open/close 8024 (time 1734677842.98 total 333.32 last 34.19) - open/close 8298 (time 1734677852.98 total 343.33 last 27.40) - open/close 8515 (time 1734677865.62 total 355.96 last 17.18) - open/close 8743 (time 1734677876.49 total 366.83 last 20.97) - open/close 9022 (time 1734677886.50 total 376.84 last 27.87) - open/close 9264 (time 1734677898.97 total 389.32 last 19.41) - open/close 10000 (time 1734677904.53 total 394.87 last 132.56) - open/close 11693 (time 1734677914.53 total 404.87 last 169.25) - open/close 13464 (time 1734677924.53 total 414.87 last 177.03) - open/close 15239 (time 1734677934.53 total 424.87 last 177.49) - open/close 17113 (time 1734677944.53 total 434.88 last 187.36) - open/close 19145 (time 1734677954.54 total 444.88 last 203.12) - open/close 20000 (time 1734677958.76 total 449.10 last 202.70) - open/close 22448 (time 1734677968.76 total 459.10 last 244.70) - open/close 24880 (time 1734677978.76 total 469.10 last 243.16) - open/close 27083 (time 1734677988.76 total 479.10 last 220.29) - open/close 29031 (time 1734677998.77 total 489.11 last 194.75) - open/close 30000 (time 1734678004.42 total 494.76 last 171.32) - open/close 31718 (time 1734678014.43 total 504.77 last 171.73) - open/close 33825 (time 1734678024.43 total 514.77 last 210.63) - open/close 35972 (time 1734678034.43 total 524.78 last 214.60) - open/close 37800 (time 1734678044.44 total 534.78 last 182.72) - open/close 39361 (time 1734678054.44 total 544.78 last 156.03) - open/close 40000 (time 1734678058.33 total 548.67 last 164.44) - open/close 41666 (time 1734678068.33 total 558.67 last 166.56) - open/close 43626 (time 1734678078.34 total 568.68 last 195.91) - open/close 45680 (time 1734678088.34 total 578.68 last 205.32) - open/close 47694 (time 1734678098.34 total 588.68 last 201.40) - open/close 49371 (time 1734678108.34 total 598.68 last 167.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 600.13 seconds: 82.70 ops/second - unlinked 0 (time 1734678111 ; total 0 ; last 0) - unlinked 10000 (time 1734678145 ; total 34 ; last 34) - unlinked 20000 (time 1734678177 ; total 66 ; last 32) - unlinked 30000 (time 1734678211 ; total 100 ; last 34) - unlinked 40000 (time 1734678245 ; total 134 ; last 34) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 164 seconds: 302.634155 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2