== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:00:22 (1788616822) Stopping clients: oleg343-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg343-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg343-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg343-server oleg343-server: oleg343-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: kmemleak disabled oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg343-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 711 (time 1788617026.35 total 10.01 last 71.00) - open/close 1431 (time 1788617036.36 total 20.03 last 71.90) - open/close 2116 (time 1788617046.37 total 30.03 last 68.48) - open/close 2842 (time 1788617056.37 total 40.04 last 72.55) - open/close 3531 (time 1788617066.39 total 50.05 last 68.80) - open/close 4260 (time 1788617076.40 total 60.06 last 72.82) total: 4500 open/close in 63.87 seconds: 70.46 ops/second - /unlink 1428 (time 1788617095.60 total 10.01 last 142.62) - /unlink 2816 (time 1788617105.60 total 20.01 last 138.77) - /unlink 4177 (time 1788617115.61 total 30.02 last 135.98) total: 4500 /unlink in 32.39 seconds: 138.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 683 (time 1788617132.53 total 10.02 last 68.17) - open/close 1447 (time 1788617142.54 total 20.03 last 76.35) - open/close 2181 (time 1788617152.54 total 30.03 last 73.39) - open/close 2921 (time 1788617162.56 total 40.05 last 73.83) - open/close 3657 (time 1788617172.57 total 50.06 last 73.53) - open/close 4437 (time 1788617182.57 total 60.06 last 77.99) total: 4500 open/close in 60.61 seconds: 74.25 ops/second - /unlink 1237 (time 1788617197.54 total 10.01 last 123.54) - /unlink 2328 (time 1788617207.54 total 20.01 last 109.10) - /unlink 3654 (time 1788617217.55 total 30.03 last 132.43) total: 4500 /unlink in 37.64 seconds: 119.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 657 (time 1788617241.34 total 10.02 last 65.57) - open/close 1243 (time 1788617251.36 total 20.04 last 58.48) - open/close 1879 (time 1788617261.37 total 30.05 last 63.55) - open/close 2571 (time 1788617271.37 total 40.05 last 69.17) - open/close 3234 (time 1788617281.38 total 50.07 last 66.21) - open/close 3821 (time 1788617291.39 total 60.07 last 58.69) - open/close 4445 (time 1788617301.40 total 70.08 last 62.33) total: 4500 open/close in 70.97 seconds: 63.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1178 (time 1788617317.06 total 10.02 last 117.61) - /unlink 2272 (time 1788617327.07 total 20.02 last 109.36) - /unlink 3680 (time 1788617337.07 total 30.02 last 140.74) total: 4500 /unlink in 36.28 seconds: 124.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 536 (time 1788617358.14 total 10.02 last 53.50) - open/close 1120 (time 1788617368.16 total 20.04 last 58.28) - open/close 1712 (time 1788617378.17 total 30.04 last 59.18) - open/close 2271 (time 1788617388.21 total 40.09 last 55.64) - open/close 2958 (time 1788617398.23 total 50.11 last 68.57) - open/close 3705 (time 1788617408.24 total 60.11 last 74.67) - open/close 4391 (time 1788617418.25 total 70.13 last 68.49) total: 4500 open/close in 71.85 seconds: 62.63 ops/second - /unlink 1431 (time 1788617434.62 total 10.00 last 143.07) - /unlink 2727 (time 1788617444.63 total 20.01 last 129.52) - /unlink 4119 (time 1788617454.63 total 30.02 last 139.10) total: 4500 /unlink in 32.75 seconds: 137.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 722 (time 1788617472.34 total 10.01 last 72.16) - open/close 1455 (time 1788617482.34 total 20.01 last 73.26) - open/close 2211 (time 1788617492.34 total 30.01 last 75.59) - open/close 2811 (time 1788617502.34 total 40.01 last 59.99) - open/close 3461 (time 1788617512.35 total 50.02 last 64.95) - open/close 4090 (time 1788617522.35 total 60.02 last 62.89) total: 4500 open/close in 65.80 seconds: 68.38 ops/second - /unlink 1385 (time 1788617543.16 total 10.00 last 138.44) - /unlink 2801 (time 1788617553.16 total 20.01 last 141.58) - /unlink 4165 (time 1788617563.17 total 30.01 last 136.32) total: 4500 /unlink in 32.78 seconds: 137.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 712 (time 1788617581.46 total 10.01 last 71.15) - open/close 1434 (time 1788617591.46 total 20.01 last 72.18) - open/close 2128 (time 1788617601.46 total 30.01 last 69.39) - open/close 2866 (time 1788617611.46 total 40.01 last 73.80) - open/close 3596 (time 1788617621.47 total 50.02 last 72.95) - open/close 4403 (time 1788617631.48 total 60.03 last 80.62) total: 4500 open/close in 61.23 seconds: 73.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1343 (time 1788617646.99 total 10.01 last 134.23) - /unlink 2679 (time 1788617656.99 total 20.01 last 133.53) - /unlink 3968 (time 1788617666.99 total 30.01 last 128.89) total: 4500 /unlink in 34.63 seconds: 129.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 705 (time 1788617686.76 total 10.00 last 70.49) - open/close 1399 (time 1788617696.77 total 20.00 last 69.38) - open/close 2002 (time 1788617706.77 total 30.01 last 60.27) - open/close 2693 (time 1788617716.78 total 40.02 last 69.04) - open/close 3452 (time 1788617726.78 total 50.02 last 75.89) - open/close 4280 (time 1788617736.79 total 60.03 last 82.71) total: 4500 open/close in 62.74 seconds: 71.72 ops/second - /unlink 1513 (time 1788617754.01 total 10.00 last 151.27) - /unlink 2955 (time 1788617764.01 total 20.01 last 144.09) - /unlink 4293 (time 1788617774.02 total 30.02 last 133.73) total: 4500 /unlink in 31.55 seconds: 142.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 759 (time 1788617789.85 total 10.01 last 75.85) - open/close 1502 (time 1788617799.86 total 20.01 last 74.27) - open/close 2188 (time 1788617809.87 total 30.03 last 68.48) - open/close 2920 (time 1788617819.88 total 40.03 last 73.18) - open/close 3632 (time 1788617829.90 total 50.05 last 71.05) - open/close 4297 (time 1788617839.92 total 60.07 last 66.37) total: 4500 open/close in 62.95 seconds: 71.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1246 (time 1788617857.19 total 10.00 last 124.59) - /unlink 2474 (time 1788617867.20 total 20.01 last 122.73) - /unlink 3812 (time 1788617877.20 total 30.01 last 133.72) total: 4500 /unlink in 35.86 seconds: 125.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 677 (time 1788617897.00 total 10.01 last 67.64) - open/close 1367 (time 1788617907.01 total 20.02 last 68.92) - open/close 2070 (time 1788617917.02 total 30.03 last 70.25) - open/close 2812 (time 1788617927.02 total 40.03 last 74.16) - open/close 3536 (time 1788617937.03 total 50.04 last 72.36) - open/close 4245 (time 1788617947.04 total 60.05 last 70.85) total: 4500 open/close in 63.72 seconds: 70.62 ops/second - /unlink 1277 (time 1788617965.01 total 10.01 last 127.58) - /unlink 2603 (time 1788617975.01 total 20.01 last 132.56) - /unlink 3951 (time 1788617985.02 total 30.02 last 134.71) total: 4500 /unlink in 34.10 seconds: 131.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 671 (time 1788618003.68 total 10.00 last 67.07) - open/close 1383 (time 1788618013.69 total 20.01 last 71.16) - open/close 2088 (time 1788618023.70 total 30.02 last 70.42) - open/close 2732 (time 1788618033.72 total 40.04 last 64.25) - open/close 3452 (time 1788618043.73 total 50.05 last 71.96) - open/close 4205 (time 1788618053.73 total 60.05 last 75.27) total: 4500 open/close in 63.93 seconds: 70.39 ops/second - /unlink 1312 (time 1788618071.98 total 10.00 last 131.14) - /unlink 2587 (time 1788618081.99 total 20.01 last 127.44) - /unlink 3905 (time 1788618091.99 total 30.01 last 131.79) total: 4500 /unlink in 34.73 seconds: 129.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 655 (time 1788618111.33 total 10.02 last 65.36) - open/close 1219 (time 1788618121.34 total 20.04 last 56.30) - open/close 1744 (time 1788618131.35 total 30.05 last 52.46) - open/close 2213 (time 1788618141.36 total 40.05 last 46.88) - open/close 2812 (time 1788618151.37 total 50.07 last 59.79) - open/close 3442 (time 1788618161.39 total 60.09 last 62.89) - open/close 4051 (time 1788618171.40 total 70.10 last 60.82) total: 4500 open/close in 77.00 seconds: 58.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1250 (time 1788618192.39 total 10.00 last 124.96) - /unlink 2583 (time 1788618202.39 total 20.00 last 133.28) - /unlink 3858 (time 1788618212.40 total 30.02 last 127.32) total: 4500 /unlink in 34.72 seconds: 129.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 753 (time 1788618231.11 total 10.01 last 75.19) - open/close 1400 (time 1788618241.12 total 20.03 last 64.61) - open/close 2125 (time 1788618251.14 total 30.05 last 72.37) - open/close 2860 (time 1788618261.16 total 40.07 last 73.34) - open/close 3564 (time 1788618271.17 total 50.08 last 70.34) - open/close 4234 (time 1788618281.17 total 60.08 last 66.97) total: 4500 open/close in 63.52 seconds: 70.84 ops/second - /unlink 1376 (time 1788618299.07 total 10.00 last 137.56) - /unlink 2741 (time 1788618309.08 total 20.01 last 136.41) - /unlink 4055 (time 1788618319.08 total 30.02 last 131.30) total: 4500 /unlink in 33.58 seconds: 134.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 753 (time 1788618338.42 total 10.04 last 75.00) - open/close 1442 (time 1788618348.44 total 20.05 last 68.81) - open/close 2217 (time 1788618358.44 total 30.06 last 77.45) - open/close 2929 (time 1788618368.47 total 40.08 last 71.03) - open/close 3653 (time 1788618378.47 total 50.09 last 72.36) - open/close 4353 (time 1788618388.47 total 60.09 last 70.00) total: 4500 open/close in 62.33 seconds: 72.20 ops/second - /unlink 1341 (time 1788618405.03 total 10.01 last 133.99) - /unlink 2685 (time 1788618415.04 total 20.02 last 134.29) - /unlink 4075 (time 1788618425.04 total 30.02 last 138.96) total: 4500 /unlink in 32.87 seconds: 136.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 719 (time 1788618442.08 total 10.01 last 71.81) - open/close 1405 (time 1788618452.09 total 20.02 last 68.55) - open/close 2088 (time 1788618462.09 total 30.02 last 68.28) - open/close 2774 (time 1788618472.09 total 40.02 last 68.60) - open/close 3420 (time 1788618482.11 total 50.03 last 64.53) - open/close 3975 (time 1788618492.12 total 60.05 last 55.40) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1224 (time 1788618515.11 total 10.02 last 122.19) - /unlink 2351 (time 1788618525.13 total 20.04 last 112.46) - /unlink 3525 (time 1788618535.14 total 30.04 last 117.35) total: 4500 /unlink in 37.94 seconds: 118.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 646 (time 1788618558.21 total 10.01 last 64.51) - open/close 1283 (time 1788618568.21 total 20.01 last 63.70) - open/close 2040 (time 1788618578.21 total 30.02 last 75.68) - open/close 2742 (time 1788618588.22 total 40.02 last 70.16) - open/close 3499 (time 1788618598.23 total 50.04 last 75.60) - open/close 4234 (time 1788618608.24 total 60.05 last 73.43) total: 4500 open/close in 63.67 seconds: 70.68 ops/second - /unlink 1320 (time 1788618626.54 total 10.00 last 131.94) - /unlink 2802 (time 1788618636.55 total 20.01 last 148.09) - /unlink 4233 (time 1788618646.55 total 30.01 last 143.10) total: 4500 /unlink in 32.29 seconds: 139.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 722 (time 1788618662.96 total 10.00 last 72.17) - open/close 1406 (time 1788618672.97 total 20.02 last 68.33) - open/close 2095 (time 1788618682.98 total 30.02 last 68.86) - open/close 2827 (time 1788618692.98 total 40.03 last 73.14) - open/close 3502 (time 1788618702.99 total 50.04 last 67.47) - open/close 4149 (time 1788618712.99 total 60.04 last 64.66) total: 4500 open/close in 64.70 seconds: 69.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1467 (time 1788618732.00 total 10.00 last 146.66) - /unlink 2649 (time 1788618742.00 total 20.01 last 118.16) - /unlink 4051 (time 1788618752.01 total 30.01 last 140.15) total: 4500 /unlink in 33.56 seconds: 134.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 689 (time 1788618770.80 total 10.01 last 68.81) - open/close 1358 (time 1788618780.81 total 20.02 last 66.85) - open/close 1974 (time 1788618790.82 total 30.03 last 61.53) - open/close 2610 (time 1788618800.84 total 40.05 last 63.48) - open/close 3238 (time 1788618810.85 total 50.06 last 62.73) - open/close 3838 (time 1788618820.88 total 60.09 last 59.82) - open/close 4438 (time 1788618830.89 total 70.10 last 59.96) total: 4500 open/close in 71.13 seconds: 63.27 ops/second - /unlink 1258 (time 1788618847.11 total 10.01 last 125.66) - /unlink 2593 (time 1788618857.12 total 20.02 last 133.37) - /unlink 3891 (time 1788618867.13 total 30.04 last 129.62) total: 4500 /unlink in 34.54 seconds: 130.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 665 (time 1788618886.17 total 10.01 last 66.46) - open/close 1216 (time 1788618896.18 total 20.02 last 55.02) - open/close 1828 (time 1788618906.19 total 30.03 last 61.14) - open/close 2324 (time 1788618916.23 total 40.07 last 49.42) - open/close 2751 (time 1788618926.24 total 50.08 last 42.65) - open/close 3339 (time 1788618936.25 total 60.09 last 58.76) - open/close 3943 (time 1788618946.25 total 70.09 last 60.35) - open/close 4491 (time 1788618956.26 total 80.10 last 54.77) total: 4500 open/close in 80.26 seconds: 56.07 ops/second - /unlink 1239 (time 1788618971.86 total 10.00 last 123.86) - /unlink 2362 (time 1788618981.87 total 20.01 last 112.22) - /unlink 3469 (time 1788618991.89 total 30.03 last 110.50) total: 4500 /unlink in 39.27 seconds: 114.60 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95293 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: kmemleak disabled modules unloaded.