== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:11:54 (1789114314) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg349-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg349-server oleg349-server: oleg349-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg349-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg349-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 842 (time 1789114499.48 total 10.00 last 84.20) - open/close 1587 (time 1789114509.48 total 20.00 last 74.49) - open/close 2382 (time 1789114519.48 total 30.00 last 79.50) - open/close 3181 (time 1789114529.49 total 40.01 last 79.87) - open/close 4011 (time 1789114539.49 total 50.01 last 82.96) total: 4500 open/close in 56.13 seconds: 80.17 ops/second - /unlink 1345 (time 1789114559.81 total 10.00 last 134.44) - /unlink 2743 (time 1789114569.81 total 20.01 last 139.77) - /unlink 4255 (time 1789114579.81 total 30.01 last 151.17) total: 4500 /unlink in 31.64 seconds: 142.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 773 (time 1789114596.40 total 10.01 last 77.20) - open/close 1560 (time 1789114606.40 total 20.01 last 78.69) - open/close 2355 (time 1789114616.41 total 30.03 last 79.39) - open/close 3163 (time 1789114626.42 total 40.03 last 80.75) - open/close 4015 (time 1789114636.42 total 50.04 last 85.18) total: 4500 open/close in 55.49 seconds: 81.10 ops/second - /unlink 1569 (time 1789114655.67 total 10.00 last 156.86) - /unlink 3014 (time 1789114665.67 total 20.01 last 144.46) total: 4500 /unlink in 29.69 seconds: 151.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 833 (time 1789114690.25 total 10.00 last 83.28) - open/close 1524 (time 1789114700.25 total 20.00 last 69.09) - open/close 2384 (time 1789114710.26 total 30.01 last 85.93) - open/close 3100 (time 1789114720.27 total 40.02 last 71.57) - open/close 3888 (time 1789114730.28 total 50.03 last 78.71) total: 4500 open/close in 57.73 seconds: 77.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1459 (time 1789114753.46 total 10.00 last 145.84) - /unlink 2994 (time 1789114763.47 total 20.01 last 153.48) - /unlink 4485 (time 1789114773.47 total 30.01 last 149.10) total: 4500 /unlink in 30.11 seconds: 149.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 734 (time 1789114788.10 total 10.00 last 73.37) - open/close 1396 (time 1789114798.11 total 20.01 last 66.14) - open/close 2130 (time 1789114808.11 total 30.02 last 73.38) - open/close 2874 (time 1789114818.12 total 40.03 last 74.33) - open/close 3680 (time 1789114828.12 total 50.03 last 80.59) - open/close 4349 (time 1789114838.14 total 60.05 last 66.78) total: 4500 open/close in 61.77 seconds: 72.85 ops/second - /unlink 1610 (time 1789114854.99 total 10.00 last 160.96) - /unlink 2781 (time 1789114864.99 total 20.00 last 117.09) - /unlink 4417 (time 1789114875.00 total 30.00 last 163.58) total: 4500 /unlink in 30.74 seconds: 146.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 911 (time 1789114890.49 total 10.02 last 90.95) - open/close 1727 (time 1789114900.51 total 20.03 last 81.47) - open/close 2521 (time 1789114910.53 total 30.05 last 79.23) - open/close 3347 (time 1789114920.53 total 40.06 last 82.54) - open/close 4170 (time 1789114930.54 total 50.06 last 82.29) total: 4500 open/close in 55.81 seconds: 80.64 ops/second - /unlink 1481 (time 1789114951.47 total 10.00 last 148.03) - /unlink 2992 (time 1789114961.48 total 20.01 last 150.95) - /unlink 4449 (time 1789114971.48 total 30.02 last 145.65) total: 4500 /unlink in 30.55 seconds: 147.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 850 (time 1789114985.95 total 10.01 last 84.94) - open/close 1684 (time 1789114995.96 total 20.01 last 83.37) - open/close 2564 (time 1789115005.96 total 30.02 last 87.95) - open/close 3438 (time 1789115015.97 total 40.02 last 87.35) - open/close 4269 (time 1789115026.01 total 50.07 last 82.74) total: 4500 open/close in 52.87 seconds: 85.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1533 (time 1789115042.60 total 10.00 last 153.25) - /unlink 2961 (time 1789115052.61 total 20.01 last 142.66) - /unlink 4265 (time 1789115062.61 total 30.02 last 130.35) total: 4500 /unlink in 31.68 seconds: 142.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 703 (time 1789115078.85 total 10.00 last 70.30) - open/close 1326 (time 1789115088.85 total 20.00 last 62.28) - open/close 1868 (time 1789115098.86 total 30.01 last 54.16) - open/close 2517 (time 1789115108.87 total 40.02 last 64.82) - open/close 3252 (time 1789115118.87 total 50.03 last 73.47) - open/close 4080 (time 1789115128.90 total 60.05 last 82.61) total: 4500 open/close in 64.81 seconds: 69.43 ops/second - /unlink 1466 (time 1789115147.44 total 10.00 last 146.57) - /unlink 2877 (time 1789115157.45 total 20.01 last 141.05) - /unlink 4373 (time 1789115167.45 total 30.01 last 149.50) total: 4500 /unlink in 30.77 seconds: 146.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 765 (time 1789115182.40 total 10.00 last 76.47) - open/close 1680 (time 1789115192.41 total 20.02 last 91.35) - open/close 2535 (time 1789115202.43 total 30.03 last 85.38) - open/close 3439 (time 1789115212.43 total 40.04 last 90.35) - open/close 4276 (time 1789115222.44 total 50.05 last 83.63) total: 4500 open/close in 52.56 seconds: 85.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1519 (time 1789115239.00 total 10.00 last 151.89) - /unlink 2963 (time 1789115249.00 total 20.00 last 144.38) total: 4500 /unlink in 29.84 seconds: 150.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 807 (time 1789115273.07 total 10.01 last 80.60) - open/close 1636 (time 1789115283.09 total 20.04 last 82.71) - open/close 2523 (time 1789115293.10 total 30.04 last 88.67) - open/close 3304 (time 1789115303.11 total 40.05 last 78.03) - open/close 3848 (time 1789115313.11 total 50.05 last 54.40) total: 4500 open/close in 58.38 seconds: 77.08 ops/second - /unlink 1361 (time 1789115335.82 total 10.00 last 136.08) - /unlink 2801 (time 1789115345.83 total 20.01 last 143.93) - /unlink 4253 (time 1789115355.83 total 30.01 last 145.17) total: 4500 /unlink in 31.56 seconds: 142.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 736 (time 1789115371.51 total 10.00 last 73.59) - open/close 1448 (time 1789115381.52 total 20.01 last 71.15) - open/close 2056 (time 1789115391.53 total 30.02 last 60.74) - open/close 2865 (time 1789115401.53 total 40.02 last 80.89) - open/close 3674 (time 1789115411.53 total 50.03 last 80.85) - open/close 4490 (time 1789115421.56 total 60.05 last 81.39) total: 4500 open/close in 60.18 seconds: 74.78 ops/second - /unlink 1454 (time 1789115436.15 total 10.00 last 145.37) - /unlink 3087 (time 1789115446.16 total 20.01 last 163.20) total: 4500 /unlink in 27.97 seconds: 160.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 805 (time 1789115469.68 total 10.01 last 80.41) - open/close 1729 (time 1789115479.68 total 20.02 last 92.35) - open/close 2552 (time 1789115489.69 total 30.02 last 82.27) - open/close 3349 (time 1789115499.70 total 40.03 last 79.62) - open/close 4159 (time 1789115509.71 total 50.04 last 80.94) total: 4500 open/close in 54.12 seconds: 83.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1649 (time 1789115527.37 total 10.01 last 164.67) - /unlink 3038 (time 1789115537.38 total 20.02 last 138.83) total: 4500 /unlink in 28.56 seconds: 157.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 825 (time 1789115559.41 total 10.01 last 82.44) - open/close 1661 (time 1789115569.44 total 20.04 last 83.33) - open/close 2457 (time 1789115579.46 total 30.06 last 79.48) - open/close 3245 (time 1789115589.46 total 40.06 last 78.79) - open/close 4029 (time 1789115599.46 total 50.06 last 78.37) total: 4500 open/close in 55.43 seconds: 81.19 ops/second - /unlink 1371 (time 1789115619.25 total 10.00 last 137.08) - /unlink 2943 (time 1789115629.26 total 20.01 last 157.05) total: 4500 /unlink in 30.00 seconds: 150.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 874 (time 1789115653.32 total 10.01 last 87.35) - open/close 1656 (time 1789115663.33 total 20.02 last 78.11) - open/close 2602 (time 1789115673.34 total 30.02 last 94.54) - open/close 3541 (time 1789115683.35 total 40.03 last 93.83) - open/close 4400 (time 1789115693.36 total 50.05 last 85.75) total: 4500 open/close in 51.05 seconds: 88.15 ops/second - /unlink 1313 (time 1789115709.93 total 10.00 last 131.28) - /unlink 2710 (time 1789115719.93 total 20.01 last 139.64) - /unlink 4114 (time 1789115729.93 total 30.01 last 140.36) total: 4500 /unlink in 33.33 seconds: 135.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 745 (time 1789115748.93 total 10.05 last 74.16) - open/close 1445 (time 1789115758.93 total 20.05 last 69.97) - open/close 2252 (time 1789115768.94 total 30.06 last 80.65) - open/close 3019 (time 1789115778.94 total 40.06 last 76.69) - open/close 3862 (time 1789115788.95 total 50.07 last 84.21) total: 4500 open/close in 57.64 seconds: 78.08 ops/second - /unlink 1580 (time 1789115809.91 total 10.01 last 157.86) - /unlink 3192 (time 1789115819.92 total 20.02 last 161.10) total: 4500 /unlink in 28.75 seconds: 156.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 883 (time 1789115843.18 total 10.00 last 88.26) - open/close 1676 (time 1789115853.18 total 20.01 last 79.29) - open/close 2545 (time 1789115863.19 total 30.01 last 86.85) - open/close 3374 (time 1789115873.19 total 40.01 last 82.88) - open/close 4262 (time 1789115883.20 total 50.03 last 88.68) total: 4500 open/close in 53.08 seconds: 84.78 ops/second - /unlink 1498 (time 1789115900.52 total 10.01 last 149.71) - /unlink 2928 (time 1789115910.53 total 20.01 last 142.95) total: 4500 /unlink in 29.95 seconds: 150.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 863 (time 1789115934.57 total 10.01 last 86.24) - open/close 1632 (time 1789115944.57 total 20.01 last 76.89) - open/close 2418 (time 1789115954.57 total 30.01 last 78.60) - open/close 3402 (time 1789115964.57 total 40.01 last 98.35) - open/close 4237 (time 1789115974.58 total 50.02 last 83.48) total: 4500 open/close in 53.04 seconds: 84.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1420 (time 1789115992.64 total 10.00 last 141.99) - /unlink 3064 (time 1789116002.64 total 20.00 last 164.34) total: 4500 /unlink in 29.61 seconds: 151.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 643 (time 1789116027.11 total 10.01 last 64.22) - open/close 1272 (time 1789116037.11 total 20.01 last 62.89) - open/close 2023 (time 1789116047.13 total 30.03 last 74.96) - open/close 2857 (time 1789116057.13 total 40.03 last 83.38) - open/close 3538 (time 1789116067.16 total 50.06 last 67.92) - open/close 4337 (time 1789116077.16 total 60.06 last 79.88) total: 4500 open/close in 62.05 seconds: 72.52 ops/second - /unlink 1449 (time 1789116092.98 total 10.00 last 144.86) - /unlink 2855 (time 1789116102.98 total 20.01 last 140.56) - /unlink 4283 (time 1789116112.99 total 30.02 last 142.64) total: 4500 /unlink in 31.71 seconds: 141.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 828 (time 1789116129.80 total 10.01 last 82.72) - open/close 1644 (time 1789116139.80 total 20.01 last 81.57) - open/close 2503 (time 1789116149.81 total 30.02 last 85.82) - open/close 3268 (time 1789116159.81 total 40.02 last 76.48) - open/close 4126 (time 1789116169.81 total 50.03 last 85.78) total: 4500 open/close in 55.16 seconds: 81.58 ops/second - /unlink 1347 (time 1789116189.57 total 10.01 last 134.62) - /unlink 2613 (time 1789116199.58 total 20.01 last 126.52) - /unlink 3961 (time 1789116209.58 total 30.01 last 134.79) total: 4500 /unlink in 33.64 seconds: 133.76 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11220: 94739 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: modules unloaded.