== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:36:21 (1734258981) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg322-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg322-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg322-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1488 (time 1734259054.76 total 10.00 last 148.79) - open/close 3016 (time 1734259064.76 total 20.00 last 152.74) - open/close 4470 (time 1734259074.77 total 30.01 last 145.32) - open/close 5866 (time 1734259084.77 total 40.01 last 139.54) - open/close 7289 (time 1734259094.78 total 50.02 last 142.25) - open/close 8703 (time 1734259104.78 total 60.02 last 141.39) - open/close 10000 (time 1734259113.63 total 68.87 last 146.47) - open/close 11790 (time 1734259123.63 total 78.87 last 178.99) - open/close 13296 (time 1734259133.64 total 88.88 last 150.55) - open/close 14765 (time 1734259143.64 total 98.88 last 146.86) - open/close 16321 (time 1734259153.65 total 108.89 last 155.51) - open/close 17894 (time 1734259163.65 total 118.89 last 157.29) - open/close 19575 (time 1734259173.65 total 128.89 last 168.10) - open/close 20000 (time 1734259176.17 total 131.41 last 168.61) - open/close 21593 (time 1734259186.17 total 141.41 last 159.30) - open/close 23075 (time 1734259196.17 total 151.41 last 148.13) - open/close 24705 (time 1734259206.17 total 161.41 last 162.96) - open/close 26101 (time 1734259216.18 total 171.42 last 139.55) - open/close 27606 (time 1734259226.18 total 181.42 last 150.49) - open/close 29167 (time 1734259236.18 total 191.42 last 156.08) - open/close 30000 (time 1734259241.42 total 196.66 last 158.94) - open/close 31940 (time 1734259251.43 total 206.67 last 193.91) - open/close 33951 (time 1734259261.43 total 216.67 last 201.06) - open/close 35610 (time 1734259271.43 total 226.67 last 165.87) - open/close 37167 (time 1734259281.44 total 236.68 last 155.57) - open/close 38976 (time 1734259291.44 total 246.68 last 180.81) - open/close 40000 (time 1734259298.38 total 253.62 last 147.68) - open/close 41537 (time 1734259308.38 total 263.62 last 153.63) - open/close 43063 (time 1734259318.38 total 273.62 last 152.57) - open/close 44799 (time 1734259328.39 total 283.63 last 173.54) - open/close 46626 (time 1734259338.39 total 293.63 last 182.61) - open/close 48376 (time 1734259348.40 total 303.64 last 174.93) - open/close 49914 (time 1734259358.40 total 313.64 last 153.78) - open/close 51440 (time 1734259368.40 total 323.64 last 152.60) - open/close 52996 (time 1734259378.40 total 333.64 last 155.58) - open/close 54519 (time 1734259388.40 total 343.64 last 152.30) - open/close 56169 (time 1734259398.40 total 353.64 last 164.96) - open/close 57771 (time 1734259408.41 total 363.65 last 160.11) - open/close 59326 (time 1734259418.41 total 373.65 last 155.47) - open/close 60000 (time 1734259422.75 total 377.99 last 155.35) - open/close 61493 (time 1734259432.76 total 388.00 last 149.16) - open/close 63038 (time 1734259442.76 total 398.00 last 154.46) - open/close 64621 (time 1734259452.76 total 408.00 last 158.26) total: 64768 open/close in 408.91 seconds: 158.39 ops/second - /unlink 2602 (time 1734259465.22 total 10.00 last 260.13) - /unlink 5226 (time 1734259475.23 total 20.01 last 262.29) - /unlink 8088 (time 1734259485.23 total 30.01 last 286.19) - /unlink 10000 (time 1734259491.92 total 36.69 last 285.91) - /unlink 12773 (time 1734259501.92 total 46.70 last 277.19) - /unlink 15349 (time 1734259511.92 total 56.70 last 257.56) - /unlink 18365 (time 1734259521.92 total 66.70 last 301.59) - /unlink 20000 (time 1734259526.97 total 71.75 last 323.89) - /unlink 23537 (time 1734259536.97 total 81.75 last 353.67) - /unlink 26421 (time 1734259546.97 total 91.75 last 288.37) - /unlink 29293 (time 1734259556.97 total 101.75 last 287.15) - /unlink 30000 (time 1734259559.13 total 103.91 last 327.46) - /unlink 32981 (time 1734259569.13 total 113.91 last 298.08) - /unlink 35605 (time 1734259579.13 total 123.91 last 262.36) - /unlink 38793 (time 1734259589.14 total 133.92 last 318.74) - /unlink 40000 (time 1734259592.73 total 137.51 last 336.18) - /unlink 43525 (time 1734259602.73 total 147.51 last 352.48) - /unlink 47299 (time 1734259612.73 total 157.51 last 377.39) - /unlink 50000 (time 1734259619.88 total 164.66 last 377.74) - /unlink 53471 (time 1734259629.88 total 174.66 last 347.04) - /unlink 57030 (time 1734259639.88 total 184.66 last 355.86) - /unlink 60000 (time 1734259648.17 total 192.95 last 358.24) - /unlink 63616 (time 1734259658.17 total 202.95 last 361.53) total: 64768 /unlink in 206.39 seconds: 313.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2076 (time 1734259688.59 total 10.00 last 207.50) - open/close 4206 (time 1734259698.59 total 20.00 last 213.00) - open/close 6519 (time 1734259708.59 total 30.01 last 231.28) - open/close 8559 (time 1734259718.59 total 40.01 last 203.98) - open/close 10000 (time 1734259725.99 total 47.40 last 194.87) - open/close 11801 (time 1734259735.99 total 57.40 last 180.05) - open/close 13650 (time 1734259745.99 total 67.41 last 184.86) - open/close 15384 (time 1734259756.00 total 77.41 last 173.30) - open/close 17175 (time 1734259766.00 total 87.41 last 179.10) - open/close 19148 (time 1734259776.00 total 97.41 last 197.29) - open/close 20000 (time 1734259780.16 total 101.57 last 204.80) - open/close 22151 (time 1734259790.16 total 111.57 last 215.08) - open/close 24083 (time 1734259800.17 total 121.58 last 193.08) - open/close 25746 (time 1734259810.17 total 131.58 last 166.23) - open/close 27613 (time 1734259820.17 total 141.59 last 186.67) - open/close 29788 (time 1734259830.18 total 151.59 last 217.39) - open/close 31913 (time 1734259840.18 total 161.59 last 212.50) - open/close 34132 (time 1734259850.18 total 171.59 last 221.82) - open/close 36316 (time 1734259860.19 total 181.60 last 218.30) - open/close 38339 (time 1734259870.19 total 191.60 last 202.24) - open/close 40000 (time 1734259879.10 total 200.52 last 186.36) - open/close 41840 (time 1734259889.11 total 210.52 last 183.88) - open/close 43580 (time 1734259899.11 total 220.52 last 173.95) - open/close 45321 (time 1734259909.11 total 230.53 last 174.08) - open/close 47324 (time 1734259919.11 total 240.53 last 200.27) - open/close 49382 (time 1734259929.12 total 250.53 last 205.74) - open/close 50000 (time 1734259932.20 total 253.61 last 200.61) - open/close 51990 (time 1734259942.20 total 263.61 last 198.92) - open/close 53957 (time 1734259952.20 total 273.62 last 196.62) - open/close 55992 (time 1734259962.21 total 283.62 last 203.49) - open/close 57954 (time 1734259972.21 total 293.62 last 196.19) - open/close 59950 (time 1734259982.21 total 303.62 last 199.59) - open/close 62059 (time 1734259992.21 total 313.62 last 210.82) - open/close 64395 (time 1734260002.21 total 323.62 last 233.57) total: 64768 open/close in 325.01 seconds: 199.28 ops/second - /unlink 4309 (time 1734260014.58 total 10.00 last 430.78) - /unlink 8332 (time 1734260024.58 total 20.00 last 402.27) - /unlink 10000 (time 1734260029.00 total 24.43 last 376.98) - /unlink 14133 (time 1734260039.00 total 34.43 last 413.29) - /unlink 17921 (time 1734260049.00 total 44.43 last 378.78) - /unlink 20000 (time 1734260053.69 total 49.11 last 443.66) - /unlink 25316 (time 1734260063.69 total 59.12 last 531.59) - /unlink 30000 (time 1734260073.19 total 68.62 last 492.96) - /unlink 34607 (time 1734260083.19 total 78.62 last 460.65) - /unlink 39005 (time 1734260093.19 total 88.62 last 439.77) - /unlink 40000 (time 1734260095.39 total 90.82 last 452.44) - /unlink 44584 (time 1734260105.39 total 100.82 last 458.31) - /unlink 49427 (time 1734260115.40 total 110.82 last 484.23) - /unlink 53783 (time 1734260125.40 total 120.82 last 435.59) - /unlink 58100 (time 1734260135.40 total 130.82 last 431.64) - /unlink 60000 (time 1734260139.94 total 135.36 last 418.32) total: 64768 /unlink in 144.91 seconds: 446.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2904 (time 1734260174.99 total 10.00 last 290.32) - open/close 5828 (time 1734260184.99 total 20.01 last 292.34) - open/close 8880 (time 1734260194.99 total 30.01 last 305.05) - open/close 10000 (time 1734260199.43 total 34.44 last 252.68) - open/close 12686 (time 1734260209.43 total 44.44 last 268.57) - open/close 15442 (time 1734260219.43 total 54.45 last 275.53) - open/close 18106 (time 1734260229.43 total 64.45 last 266.35) - open/close 20000 (time 1734260236.73 total 71.74 last 259.60) - open/close 22512 (time 1734260246.73 total 81.74 last 251.18) - open/close 25342 (time 1734260256.73 total 91.75 last 282.91) - open/close 28197 (time 1734260266.73 total 101.75 last 285.45) - open/close 30000 (time 1734260274.20 total 109.22 last 241.33) - open/close 32258 (time 1734260284.21 total 119.22 last 225.77) - open/close 34661 (time 1734260294.21 total 129.23 last 240.22) - open/close 36797 (time 1734260304.21 total 139.23 last 213.56) - open/close 39205 (time 1734260314.21 total 149.23 last 240.73) - open/close 40000 (time 1734260317.68 total 152.69 last 229.50) - open/close 42499 (time 1734260327.68 total 162.69 last 249.88) - open/close 44942 (time 1734260337.68 total 172.70 last 244.18) - open/close 47378 (time 1734260347.68 total 182.70 last 243.57) - open/close 49945 (time 1734260357.69 total 192.70 last 256.65) - open/close 52330 (time 1734260367.69 total 202.70 last 238.45) - open/close 54714 (time 1734260377.69 total 212.71 last 238.37) - open/close 56912 (time 1734260387.69 total 222.71 last 219.75) - open/close 59105 (time 1734260397.69 total 232.71 last 219.28) - open/close 60000 (time 1734260401.87 total 236.88 last 214.37) - open/close 61994 (time 1734260411.87 total 246.89 last 199.31) - open/close 64222 (time 1734260421.87 total 256.89 last 222.79) total: 64768 open/close in 259.39 seconds: 249.69 ops/second - /unlink 3786 (time 1734260435.54 total 10.00 last 378.60) - /unlink 7446 (time 1734260445.54 total 20.00 last 365.93) - /unlink 10000 (time 1734260452.01 total 26.47 last 394.67) - /unlink 14057 (time 1734260462.02 total 36.48 last 405.61) - /unlink 18277 (time 1734260472.02 total 46.48 last 421.94) - /unlink 20000 (time 1734260476.00 total 50.46 last 432.39) - /unlink 24822 (time 1734260486.00 total 60.46 last 482.15) - /unlink 29251 (time 1734260496.01 total 70.46 last 442.87) - /unlink 33916 (time 1734260506.01 total 80.46 last 466.49) - /unlink 38528 (time 1734260516.01 total 90.46 last 461.16) - /unlink 40000 (time 1734260519.06 total 93.52 last 482.49) - /unlink 44506 (time 1734260529.06 total 103.52 last 450.47) - /unlink 49222 (time 1734260539.06 total 113.52 last 471.50) - /unlink 50000 (time 1734260541.07 total 115.53 last 387.58) - /unlink 53784 (time 1734260551.07 total 125.53 last 378.32) - /unlink 57858 (time 1734260561.07 total 135.53 last 407.34) - /unlink 60000 (time 1734260566.42 total 140.88 last 400.29) - /unlink 64600 (time 1734260576.43 total 150.88 last 459.92) total: 64768 /unlink in 151.26 seconds: 428.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2597 (time 1734260605.09 total 10.00 last 259.64) - open/close 5094 (time 1734260615.10 total 20.00 last 249.66) - open/close 7934 (time 1734260625.10 total 30.01 last 283.92) - open/close 10000 (time 1734260633.54 total 38.45 last 244.63) - open/close 12339 (time 1734260643.55 total 48.46 last 233.82) - open/close 15040 (time 1734260653.55 total 58.46 last 270.04) - open/close 17920 (time 1734260663.55 total 68.46 last 288.00) - open/close 20000 (time 1734260671.55 total 76.46 last 260.02) - open/close 22553 (time 1734260681.55 total 86.46 last 255.27) - open/close 25267 (time 1734260691.56 total 96.47 last 271.14) - open/close 27926 (time 1734260701.56 total 106.47 last 265.81) - open/close 30000 (time 1734260709.54 total 114.45 last 260.01) - open/close 32998 (time 1734260719.54 total 124.45 last 299.74) - open/close 36099 (time 1734260729.55 total 134.45 last 310.02) - open/close 39164 (time 1734260739.55 total 144.45 last 306.49) - open/close 40000 (time 1734260742.13 total 147.03 last 323.83) - open/close 43537 (time 1734260752.13 total 157.04 last 353.58) - open/close 46640 (time 1734260762.14 total 167.05 last 309.92) - open/close 50000 (time 1734260771.10 total 176.01 last 375.00) - open/close 54699 (time 1734260781.10 total 186.01 last 469.83) - open/close 59037 (time 1734260791.11 total 196.01 last 433.74) - open/close 64447 (time 1734260801.11 total 206.01 last 540.90) total: 64768 open/close in 206.79 seconds: 313.20 ops/second - /unlink 7402 (time 1734260812.80 total 10.00 last 740.20) - /unlink 10000 (time 1734260816.22 total 13.42 last 760.16) - /unlink 17707 (time 1734260826.22 total 23.42 last 770.66) - /unlink 20000 (time 1734260829.21 total 26.41 last 767.38) - /unlink 28035 (time 1734260839.21 total 36.41 last 803.44) - /unlink 30000 (time 1734260841.83 total 39.03 last 748.26) - /unlink 37192 (time 1734260851.84 total 49.03 last 719.15) - /unlink 40000 (time 1734260855.51 total 52.70 last 765.11) - /unlink 48039 (time 1734260865.51 total 62.70 last 803.90) - /unlink 50000 (time 1734260867.75 total 64.95 last 872.28) - /unlink 59590 (time 1734260877.75 total 74.95 last 958.92) total: 64768 /unlink in 80.35 seconds: 806.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4420 (time 1734260920.83 total 10.00 last 441.96) - open/close 9270 (time 1734260930.84 total 20.00 last 484.95) - open/close 14796 (time 1734260940.84 total 30.00 last 552.58) - open/close 20000 (time 1734260950.20 total 39.37 last 555.69) - open/close 25062 (time 1734260960.20 total 49.37 last 506.17) - open/close 29483 (time 1734260970.20 total 59.37 last 442.01) - open/close 34177 (time 1734260980.20 total 69.37 last 469.37) - open/close 38888 (time 1734260990.27 total 79.44 last 467.84) - open/close 40000 (time 1734260992.59 total 81.75 last 480.77) - open/close 44857 (time 1734261002.59 total 91.75 last 485.67) - open/close 49634 (time 1734261012.59 total 101.76 last 477.60) - open/close 54261 (time 1734261022.59 total 111.76 last 462.62) - open/close 58635 (time 1734261032.59 total 121.76 last 437.34) - open/close 60000 (time 1734261035.40 total 124.56 last 486.47) total: 64768 open/close in 134.35 seconds: 482.09 ops/second - /unlink 8311 (time 1734261056.05 total 10.00 last 831.08) - /unlink 10000 (time 1734261058.10 total 12.05 last 826.01) - /unlink 18363 (time 1734261068.10 total 22.05 last 836.23) - /unlink 20000 (time 1734261070.10 total 24.05 last 818.33) - /unlink 28316 (time 1734261080.10 total 34.05 last 831.60) - /unlink 36826 (time 1734261090.10 total 44.05 last 850.92) - /unlink 40000 (time 1734261093.77 total 47.72 last 863.83) - /unlink 48179 (time 1734261103.77 total 57.72 last 817.85) - /unlink 50000 (time 1734261105.99 total 59.94 last 821.21) - /unlink 58264 (time 1734261115.99 total 69.94 last 826.37) - /unlink 60000 (time 1734261118.07 total 72.02 last 834.13) total: 64768 /unlink in 78.71 seconds: 822.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2