== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:29:11 (1744140551) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5338 (time 1744140585.03 total 10.00 last 533.75) - open/close 10000 (time 1744140593.26 total 18.23 last 566.46) - open/close 15249 (time 1744140603.26 total 28.23 last 524.79) - open/close 20000 (time 1744140611.23 total 36.20 last 596.03) - open/close 26005 (time 1744140621.23 total 46.20 last 600.45) - open/close 30000 (time 1744140627.75 total 52.72 last 613.48) - open/close 35383 (time 1744140637.75 total 62.72 last 538.24) - open/close 40000 (time 1744140646.05 total 71.03 last 555.74) - open/close 45653 (time 1744140656.06 total 81.03 last 565.27) - open/close 50000 (time 1744140663.41 total 88.38 last 590.74) - open/close 55827 (time 1744140673.41 total 98.39 last 582.65) - open/close 60000 (time 1744140680.77 total 105.74 last 567.60) total: 64768 open/close in 114.09 seconds: 567.69 ops/second - /unlink 9155 (time 1744140699.92 total 10.00 last 915.44) - /unlink 18634 (time 1744140709.92 total 20.00 last 947.79) - /unlink 27980 (time 1744140719.93 total 30.00 last 934.53) - /unlink 30000 (time 1744140722.20 total 32.28 last 887.44) - /unlink 38337 (time 1744140732.20 total 42.28 last 833.70) - /unlink 46576 (time 1744140742.20 total 52.28 last 823.89) - /unlink 50000 (time 1744140745.93 total 56.01 last 917.75) - /unlink 58701 (time 1744140755.93 total 66.01 last 870.07) total: 64768 /unlink in 73.06 seconds: 886.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5170 (time 1744140798.16 total 10.00 last 517.00) - open/close 10000 (time 1744140807.12 total 18.96 last 539.10) - open/close 15275 (time 1744140817.12 total 28.96 last 527.42) - open/close 20000 (time 1744140826.21 total 38.05 last 519.68) - open/close 24585 (time 1744140836.21 total 48.05 last 458.43) - open/close 29300 (time 1744140846.22 total 58.06 last 471.43) - open/close 34443 (time 1744140856.22 total 68.06 last 514.22) - open/close 39649 (time 1744140866.22 total 78.06 last 520.53) - open/close 44838 (time 1744140876.22 total 88.06 last 518.86) - open/close 49991 (time 1744140886.22 total 98.06 last 515.26) - open/close 55133 (time 1744140896.22 total 108.06 last 514.16) - open/close 59995 (time 1744140906.22 total 118.06 last 486.13) - open/close 64585 (time 1744140916.22 total 128.06 last 458.95) total: 64768 open/close in 128.56 seconds: 503.78 ops/second - /unlink 8058 (time 1744140927.61 total 10.00 last 805.76) - /unlink 10000 (time 1744140930.13 total 12.52 last 769.91) - /unlink 18354 (time 1744140940.14 total 22.52 last 835.31) - /unlink 20000 (time 1744140942.28 total 24.67 last 768.30) - /unlink 28037 (time 1744140952.28 total 34.67 last 803.63) - /unlink 30000 (time 1744140954.55 total 36.94 last 862.58) - /unlink 37974 (time 1744140964.56 total 46.94 last 797.33) - /unlink 40000 (time 1744140967.16 total 49.54 last 778.95) - /unlink 48278 (time 1744140977.16 total 59.55 last 827.71) - /unlink 50000 (time 1744140979.19 total 61.58 last 846.82) - /unlink 58250 (time 1744140989.19 total 71.58 last 824.99) total: 64768 /unlink in 78.89 seconds: 820.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5489 (time 1744141028.75 total 10.00 last 548.80) - open/close 10000 (time 1744141037.19 total 18.43 last 535.08) - open/close 15287 (time 1744141047.19 total 28.43 last 528.66) - open/close 20000 (time 1744141056.04 total 37.29 last 532.08) - open/close 24932 (time 1744141066.04 total 47.29 last 493.19) - open/close 30000 (time 1744141075.77 total 57.02 last 521.00) - open/close 35196 (time 1744141085.77 total 67.02 last 519.59) - open/close 40000 (time 1744141095.23 total 76.47 last 508.15) - open/close 45223 (time 1744141105.23 total 86.47 last 522.24) - open/close 50000 (time 1744141114.22 total 95.46 last 531.39) - open/close 55276 (time 1744141124.22 total 105.46 last 527.55) - open/close 60000 (time 1744141133.60 total 114.84 last 503.70) total: 64768 open/close in 124.30 seconds: 521.05 ops/second - /unlink 8852 (time 1744141153.94 total 10.00 last 885.17) - /unlink 17781 (time 1744141163.94 total 20.00 last 892.82) - /unlink 20000 (time 1744141166.43 total 22.49 last 891.51) - /unlink 28754 (time 1744141176.43 total 32.49 last 875.32) - /unlink 37249 (time 1744141186.43 total 42.49 last 849.43) - /unlink 40000 (time 1744141189.70 total 45.76 last 842.96) - /unlink 48822 (time 1744141199.70 total 55.76 last 882.17) - /unlink 57700 (time 1744141209.70 total 65.76 last 887.80) - /unlink 60000 (time 1744141213.11 total 69.17 last 673.23) total: 64768 /unlink in 74.75 seconds: 866.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3842 (time 1744141254.10 total 10.00 last 384.13) - open/close 8518 (time 1744141264.10 total 20.00 last 467.60) - open/close 10000 (time 1744141267.15 total 23.05 last 486.19) - open/close 14480 (time 1744141277.15 total 33.05 last 447.91) - open/close 19032 (time 1744141287.15 total 43.05 last 455.12) - open/close 23303 (time 1744141297.15 total 53.06 last 427.01) - open/close 26658 (time 1744141307.15 total 63.06 last 335.45) - open/close 30000 (time 1744141314.23 total 70.13 last 472.27) - open/close 35195 (time 1744141324.23 total 80.13 last 519.46) - open/close 40000 (time 1744141334.20 total 90.11 last 481.87) - open/close 44805 (time 1744141344.20 total 100.11 last 480.49) - open/close 49669 (time 1744141354.21 total 110.11 last 486.30) - open/close 54602 (time 1744141364.21 total 120.11 last 493.21) - open/close 59473 (time 1744141374.21 total 130.11 last 487.03) - open/close 63732 (time 1744141384.21 total 140.11 last 425.90) total: 64768 open/close in 143.20 seconds: 452.28 ops/second - /unlink 7162 (time 1744141398.45 total 10.00 last 716.18) - /unlink 10000 (time 1744141403.67 total 15.22 last 543.90) - /unlink 16692 (time 1744141413.67 total 25.22 last 669.07) - /unlink 20000 (time 1744141418.57 total 30.12 last 675.05) - /unlink 27144 (time 1744141428.57 total 40.12 last 714.34) - /unlink 30000 (time 1744141431.46 total 43.01 last 987.10) - /unlink 38731 (time 1744141441.46 total 53.02 last 873.03) - /unlink 46016 (time 1744141451.46 total 63.02 last 728.45) - /unlink 50000 (time 1744141456.96 total 68.51 last 724.80) - /unlink 58436 (time 1744141466.96 total 78.51 last 843.53) total: 64768 /unlink in 85.57 seconds: 756.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4049 (time 1744141505.27 total 10.00 last 404.88) - open/close 8293 (time 1744141515.27 total 20.00 last 424.36) - open/close 10000 (time 1744141519.44 total 24.17 last 409.85) - open/close 14271 (time 1744141529.44 total 34.17 last 427.06) - open/close 18310 (time 1744141539.44 total 44.17 last 403.89) - open/close 20000 (time 1744141543.27 total 48.00 last 440.48) - open/close 23780 (time 1744141553.28 total 58.01 last 377.86) - open/close 27285 (time 1744141563.28 total 68.01 last 350.49) - open/close 30000 (time 1744141571.06 total 75.79 last 349.01) - open/close 34096 (time 1744141581.06 total 85.79 last 409.59) - open/close 37921 (time 1744141591.06 total 95.79 last 382.48) - open/close 40000 (time 1744141596.83 total 101.56 last 360.39) - open/close 43895 (time 1744141606.83 total 111.56 last 389.46) - open/close 47227 (time 1744141616.83 total 121.56 last 333.18) - open/close 50000 (time 1744141625.50 total 130.23 last 319.79) - open/close 53349 (time 1744141635.50 total 140.23 last 334.88) - open/close 56102 (time 1744141645.50 total 150.23 last 275.27) - open/close 58809 (time 1744141655.50 total 160.23 last 270.69) - open/close 60000 (time 1744141660.02 total 164.75 last 263.79) - open/close 62647 (time 1744141670.02 total 174.75 last 264.62) total: 64768 open/close in 183.13 seconds: 353.68 ops/second - /unlink 4944 (time 1744141689.66 total 10.00 last 494.36) - /unlink 9870 (time 1744141699.66 total 20.00 last 492.59) - /unlink 14861 (time 1744141709.66 total 30.00 last 499.04) - /unlink 20000 (time 1744141719.13 total 39.47 last 542.98) - /unlink 24865 (time 1744141729.13 total 49.47 last 486.50) - /unlink 29672 (time 1744141739.13 total 59.47 last 480.64) - /unlink 34897 (time 1744141749.13 total 69.47 last 522.46) - /unlink 39738 (time 1744141759.13 total 79.47 last 484.05) - /unlink 44671 (time 1744141769.13 total 89.47 last 493.22) - /unlink 50000 (time 1744141777.22 total 97.56 last 659.02) - /unlink 56639 (time 1744141787.22 total 107.56 last 663.78) - /unlink 60000 (time 1744141796.70 total 117.04 last 354.57) total: 64768 /unlink in 124.76 seconds: 519.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2