== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:48:06 (1788328086)
Stopping clients: oleg610-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg610-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg610-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg610-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg610-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg610-server
oleg610-server: oleg610-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg610-server'
oleg610-server: oleg610-server.virtnet: executing load_modules_local
oleg610-server: Loading modules from /home/green/git/lustre-release/lustre
oleg610-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg610-server: detected 4 online CPUs by sysfs
oleg610-server: MODOPTS_LIBCFS=
oleg610-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg610-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all
pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg610-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all
pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg610-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all
pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg610-client.virtnet:  -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 708 (time 1788328284.93 total 10.01 last 70.72)
 - open/close 1333 (time 1788328294.93 total 20.01 last 62.49)
 - open/close 1870 (time 1788328304.93 total 30.02 last 53.68)
 - open/close 2501 (time 1788328314.94 total 40.02 last 63.07)
 - open/close 3185 (time 1788328324.95 total 50.03 last 68.31)
 - open/close 3857 (time 1788328334.97 total 60.05 last 67.10)
 - open/close 4452 (time 1788328345.04 total 70.12 last 59.07)
total: 4500 open/close in 70.98 seconds: 63.40 ops/second
 - /unlink 1285 (time 1788328360.78 total 10.00 last 128.48)
 - /unlink 2699 (time 1788328370.81 total 20.03 last 141.01)
 - /unlink 3991 (time 1788328380.82 total 30.04 last 129.07)
total: 4500 /unlink in 33.93 seconds: 132.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 710 (time 1788328398.99 total 10.00 last 70.97)
 - open/close 1328 (time 1788328408.99 total 20.01 last 61.78)
 - open/close 1951 (time 1788328419.00 total 30.01 last 62.27)
 - open/close 2450 (time 1788328429.01 total 40.02 last 49.84)
 - open/close 3036 (time 1788328439.02 total 50.03 last 58.55)
 - open/close 3704 (time 1788328449.05 total 60.06 last 66.59)
 - open/close 4336 (time 1788328459.06 total 70.07 last 63.13)
total: 4500 open/close in 72.61 seconds: 61.98 ops/second
 - /unlink 1115 (time 1788328476.80 total 10.01 last 111.39)
 - /unlink 2066 (time 1788328486.80 total 20.01 last 95.08)
 - /unlink 3184 (time 1788328496.80 total 30.01 last 111.77)
 - /unlink 4278 (time 1788328506.81 total 40.02 last 109.36)
total: 4500 /unlink in 41.99 seconds: 107.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 545 (time 1788328524.61 total 10.01 last 54.47)
 - open/close 1109 (time 1788328534.62 total 20.02 last 56.33)
 - open/close 1790 (time 1788328544.62 total 30.02 last 68.08)
 - open/close 2502 (time 1788328554.64 total 40.04 last 71.09)
 - open/close 3111 (time 1788328564.64 total 50.04 last 60.90)
 - open/close 3818 (time 1788328574.64 total 60.04 last 70.69)
 - open/close 4401 (time 1788328584.66 total 70.05 last 58.21)
total: 4500 open/close in 71.54 seconds: 62.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1308 (time 1788328600.53 total 10.00 last 130.74)
 - /unlink 2559 (time 1788328610.53 total 20.01 last 125.08)
 - /unlink 3915 (time 1788328620.54 total 30.01 last 135.55)
total: 4500 /unlink in 34.02 seconds: 132.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 688 (time 1788328640.13 total 10.00 last 68.79)
 - open/close 1426 (time 1788328650.14 total 20.01 last 73.76)
 - open/close 2138 (time 1788328660.14 total 30.01 last 71.14)
 - open/close 2760 (time 1788328670.15 total 40.02 last 62.19)
 - open/close 3514 (time 1788328680.15 total 50.02 last 75.39)
 - open/close 4160 (time 1788328690.16 total 60.03 last 64.52)
total: 4500 open/close in 64.95 seconds: 69.28 ops/second
 - /unlink 1259 (time 1788328709.68 total 10.01 last 125.72)
 - /unlink 2564 (time 1788328719.68 total 20.02 last 130.49)
 - /unlink 3839 (time 1788328729.69 total 30.02 last 127.48)
total: 4500 /unlink in 34.86 seconds: 129.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 624 (time 1788328749.56 total 10.00 last 62.39)
 - open/close 1227 (time 1788328759.57 total 20.00 last 60.29)
 - open/close 1860 (time 1788328769.60 total 30.03 last 63.11)
 - open/close 2434 (time 1788328779.61 total 40.04 last 57.35)
 - open/close 3046 (time 1788328789.62 total 50.06 last 61.09)
 - open/close 3837 (time 1788328799.64 total 60.07 last 79.00)
total: 4500 open/close in 68.54 seconds: 65.66 ops/second
 - /unlink 1464 (time 1788328822.17 total 10.00 last 146.37)
 - /unlink 2892 (time 1788328832.18 total 20.01 last 142.74)
 - /unlink 4246 (time 1788328842.18 total 30.01 last 135.34)
total: 4500 /unlink in 32.17 seconds: 139.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 606 (time 1788328859.99 total 10.02 last 60.51)
 - open/close 1278 (time 1788328870.00 total 20.03 last 67.07)
 - open/close 2014 (time 1788328880.01 total 30.04 last 73.56)
 - open/close 2666 (time 1788328890.02 total 40.05 last 65.13)
 - open/close 3384 (time 1788328900.02 total 50.05 last 71.78)
 - open/close 4184 (time 1788328910.03 total 60.06 last 79.94)
total: 4500 open/close in 64.75 seconds: 69.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1273 (time 1788328929.97 total 10.01 last 127.12)
 - /unlink 2582 (time 1788328939.97 total 20.02 last 130.81)
 - /unlink 3813 (time 1788328950.00 total 30.04 last 122.81)
total: 4500 /unlink in 34.91 seconds: 128.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 708 (time 1788328969.92 total 10.00 last 70.78)
 - open/close 1441 (time 1788328979.92 total 20.01 last 73.27)
 - open/close 2197 (time 1788328989.93 total 30.02 last 75.49)
 - open/close 2895 (time 1788328999.95 total 40.03 last 69.73)
 - open/close 3681 (time 1788329009.97 total 50.06 last 78.37)
total: 4500 open/close in 59.86 seconds: 75.18 ops/second
 - /unlink 1438 (time 1788329034.39 total 10.01 last 143.70)
 - /unlink 2778 (time 1788329044.40 total 20.01 last 133.95)
 - /unlink 4004 (time 1788329054.40 total 30.02 last 122.53)
total: 4500 /unlink in 34.77 seconds: 129.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 686 (time 1788329074.84 total 10.04 last 68.33)
 - open/close 1296 (time 1788329084.84 total 20.04 last 61.00)
 - open/close 2071 (time 1788329094.85 total 30.05 last 77.43)
 - open/close 2868 (time 1788329104.85 total 40.05 last 79.69)
 - open/close 3537 (time 1788329114.86 total 50.06 last 66.82)
 - open/close 4346 (time 1788329124.89 total 60.10 last 80.63)
total: 4500 open/close in 62.57 seconds: 71.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1412 (time 1788329141.95 total 10.00 last 141.16)
 - /unlink 2923 (time 1788329151.95 total 20.00 last 151.08)
 - /unlink 4353 (time 1788329161.96 total 30.01 last 142.99)
total: 4500 /unlink in 31.20 seconds: 144.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 787 (time 1788329178.55 total 10.01 last 78.61)
 - open/close 1631 (time 1788329188.56 total 20.03 last 84.26)
 - open/close 2227 (time 1788329198.57 total 30.04 last 59.54)
 - open/close 2923 (time 1788329208.58 total 40.04 last 69.56)
 - open/close 3654 (time 1788329218.60 total 50.06 last 72.96)
 - open/close 4415 (time 1788329228.60 total 60.07 last 76.06)
total: 4500 open/close in 61.27 seconds: 73.44 ops/second
 - /unlink 1385 (time 1788329244.85 total 10.02 last 138.25)
 - /unlink 2854 (time 1788329254.85 total 20.02 last 146.90)
 - /unlink 4337 (time 1788329264.85 total 30.02 last 148.24)
total: 4500 /unlink in 31.55 seconds: 142.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 719 (time 1788329281.64 total 10.00 last 71.88)
 - open/close 1421 (time 1788329291.66 total 20.02 last 70.05)
 - open/close 2028 (time 1788329301.66 total 30.03 last 60.67)
 - open/close 2675 (time 1788329311.66 total 40.03 last 64.68)
 - open/close 3259 (time 1788329321.70 total 50.06 last 58.21)
 - open/close 3820 (time 1788329331.70 total 60.06 last 56.09)
total: 4500 open/close in 69.67 seconds: 64.59 ops/second
 - /unlink 1485 (time 1788329355.80 total 10.00 last 148.49)
 - /unlink 2941 (time 1788329365.80 total 20.00 last 145.55)
 - /unlink 4465 (time 1788329375.81 total 30.00 last 152.39)
total: 4500 /unlink in 30.22 seconds: 148.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 781 (time 1788329390.32 total 10.00 last 78.08)
 - open/close 1617 (time 1788329400.33 total 20.01 last 83.54)
 - open/close 2403 (time 1788329410.34 total 30.02 last 78.54)
 - open/close 3239 (time 1788329420.35 total 40.03 last 83.52)
 - open/close 4084 (time 1788329430.35 total 50.03 last 84.43)
total: 4500 open/close in 55.96 seconds: 80.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1426 (time 1788329451.73 total 10.00 last 142.55)
 - /unlink 2839 (time 1788329461.74 total 20.01 last 141.16)
 - /unlink 4315 (time 1788329471.75 total 30.02 last 147.56)
total: 4500 /unlink in 31.43 seconds: 143.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 876 (time 1788329487.44 total 10.01 last 87.51)
 - open/close 1598 (time 1788329497.44 total 20.01 last 72.19)
 - open/close 2296 (time 1788329507.45 total 30.02 last 69.74)
 - open/close 3079 (time 1788329517.45 total 40.02 last 78.27)
 - open/close 3705 (time 1788329527.46 total 50.03 last 62.56)
 - open/close 4444 (time 1788329537.46 total 60.04 last 73.85)
total: 4500 open/close in 60.71 seconds: 74.12 ops/second
 - /unlink 1363 (time 1788329553.57 total 10.01 last 136.17)
 - /unlink 2737 (time 1788329563.58 total 20.02 last 137.31)
 - /unlink 4293 (time 1788329573.58 total 30.02 last 155.57)
total: 4500 /unlink in 31.48 seconds: 142.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 720 (time 1788329589.47 total 10.00 last 71.98)
 - open/close 1499 (time 1788329599.47 total 20.01 last 77.85)
 - open/close 2192 (time 1788329609.50 total 30.03 last 69.14)
 - open/close 2853 (time 1788329619.51 total 40.04 last 66.04)
 - open/close 3550 (time 1788329629.52 total 50.05 last 69.62)
 - open/close 4305 (time 1788329639.55 total 60.08 last 75.27)
total: 4500 open/close in 62.62 seconds: 71.86 ops/second
 - /unlink 1502 (time 1788329656.55 total 10.00 last 150.17)
 - /unlink 2972 (time 1788329666.56 total 20.02 last 146.80)
total: 4500 /unlink in 29.81 seconds: 150.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 830 (time 1788329690.32 total 10.00 last 82.97)
 - open/close 1500 (time 1788329700.34 total 20.03 last 66.86)
 - open/close 2375 (time 1788329710.34 total 30.03 last 87.45)
 - open/close 3045 (time 1788329720.36 total 40.05 last 66.88)
 - open/close 3836 (time 1788329730.37 total 50.06 last 79.01)
total: 4500 open/close in 59.11 seconds: 76.13 ops/second
 - /unlink 1344 (time 1788329754.32 total 10.02 last 134.12)
 - /unlink 2589 (time 1788329764.32 total 20.02 last 124.49)
 - /unlink 4143 (time 1788329774.32 total 30.02 last 155.37)
total: 4500 /unlink in 32.52 seconds: 138.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 819 (time 1788329791.48 total 10.01 last 81.82)
 - open/close 1525 (time 1788329801.49 total 20.02 last 70.56)
 - open/close 2248 (time 1788329811.50 total 30.03 last 72.19)
 - open/close 3067 (time 1788329821.50 total 40.03 last 81.88)
 - open/close 3933 (time 1788329831.52 total 50.05 last 86.44)
total: 4500 open/close in 57.93 seconds: 77.68 ops/second
 - /unlink 1393 (time 1788329853.68 total 10.00 last 139.24)
 - /unlink 2681 (time 1788329863.68 total 20.01 last 128.75)
 - /unlink 4034 (time 1788329873.70 total 30.02 last 135.09)
total: 4500 /unlink in 33.45 seconds: 134.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 674 (time 1788329892.82 total 10.01 last 67.37)
 - open/close 1285 (time 1788329902.83 total 20.02 last 61.04)
 - open/close 1936 (time 1788329912.84 total 30.02 last 65.09)
 - open/close 2616 (time 1788329922.84 total 40.02 last 67.99)
 - open/close 3334 (time 1788329932.90 total 50.08 last 71.38)
 - open/close 4082 (time 1788329942.91 total 60.09 last 74.69)
total: 4500 open/close in 65.78 seconds: 68.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1326 (time 1788329963.50 total 10.00 last 132.55)
 - /unlink 2589 (time 1788329973.51 total 20.01 last 126.20)
 - /unlink 3759 (time 1788329983.51 total 30.01 last 116.97)
total: 4500 /unlink in 36.16 seconds: 124.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 771 (time 1788330004.58 total 10.00 last 77.08)
 - open/close 1497 (time 1788330014.59 total 20.01 last 72.56)
 - open/close 2254 (time 1788330024.59 total 30.01 last 75.70)
 - open/close 2829 (time 1788330034.60 total 40.02 last 57.45)
 - open/close 3432 (time 1788330044.61 total 50.03 last 60.25)
 - open/close 4174 (time 1788330054.61 total 60.03 last 74.20)
total: 4500 open/close in 65.21 seconds: 69.01 ops/second
 - /unlink 1312 (time 1788330074.65 total 10.01 last 131.13)
 - /unlink 2490 (time 1788330084.65 total 20.01 last 117.78)
 - /unlink 3605 (time 1788330094.66 total 30.01 last 111.47)
total: 4500 /unlink in 37.86 seconds: 118.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 573 (time 1788330117.75 total 10.01 last 57.26)
 - open/close 1223 (time 1788330127.77 total 20.02 last 64.89)
 - open/close 2049 (time 1788330137.77 total 30.03 last 82.55)
 - open/close 2780 (time 1788330147.78 total 40.03 last 73.07)
 - open/close 3397 (time 1788330157.78 total 50.03 last 61.70)
 - open/close 4003 (time 1788330167.80 total 60.05 last 60.48)
total: 4500 open/close in 66.97 seconds: 67.20 ops/second
 - /unlink 1307 (time 1788330189.61 total 10.00 last 130.67)
 - /unlink 2558 (time 1788330199.62 total 20.00 last 125.08)
 - /unlink 3849 (time 1788330209.62 total 30.01 last 129.07)
total: 4500 /unlink in 34.81 seconds: 129.26 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95294 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg610-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg610-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server
stop mds service on oleg610-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg610-server
stop mds service on oleg610-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg610-server

unloading modules via unload_modules_local on: 'oleg610-server'
oleg610-server: oleg610-server.virtnet: executing unload_modules_local
oleg610-server: 
modules unloaded.