== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:25:13 (1775780713)
Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg112-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server
oleg112-server: oleg112-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg112-server'
oleg112-server: oleg112-server.virtnet: executing load_modules_local
oleg112-server: Loading modules from /home/green/git/lustre-release/lustre
oleg112-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg112-server: detected 4 online CPUs by sysfs
oleg112-server: MODOPTS_LIBCFS=
oleg112-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg112-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all
pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg112-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all
pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg112-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all
pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg112-client.virtnet:  -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 834 (time 1775780918.43 total 10.01 last 83.34)
 - open/close 1506 (time 1775780928.44 total 20.01 last 67.18)
 - open/close 2262 (time 1775780938.44 total 30.02 last 75.53)
 - open/close 2933 (time 1775780948.45 total 40.02 last 67.09)
 - open/close 3572 (time 1775780958.46 total 50.03 last 63.81)
 - open/close 4345 (time 1775780968.46 total 60.04 last 77.28)
total: 4500 open/close in 61.83 seconds: 72.78 ops/second
 - /unlink 1314 (time 1775780985.14 total 10.01 last 131.28)
 - /unlink 2545 (time 1775780995.16 total 20.02 last 122.94)
 - /unlink 3873 (time 1775781005.16 total 30.03 last 132.75)
total: 4500 /unlink in 34.41 seconds: 130.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 576 (time 1775781025.48 total 10.01 last 57.52)
 - open/close 1303 (time 1775781035.48 total 20.02 last 72.66)
 - open/close 2015 (time 1775781045.49 total 30.03 last 71.15)
 - open/close 2816 (time 1775781055.50 total 40.04 last 80.01)
 - open/close 3437 (time 1775781065.52 total 50.06 last 61.97)
 - open/close 4102 (time 1775781075.55 total 60.09 last 66.29)
total: 4500 open/close in 66.35 seconds: 67.83 ops/second
 - /unlink 1286 (time 1775781097.08 total 10.00 last 128.57)
 - /unlink 2488 (time 1775781107.08 total 20.00 last 120.18)
 - /unlink 3905 (time 1775781117.08 total 30.01 last 141.65)
total: 4500 /unlink in 34.18 seconds: 131.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 745 (time 1775781137.31 total 10.00 last 74.50)
 - open/close 1310 (time 1775781147.32 total 20.01 last 56.46)
 - open/close 1984 (time 1775781157.33 total 30.02 last 67.34)
 - open/close 2740 (time 1775781167.33 total 40.02 last 75.56)
 - open/close 3503 (time 1775781177.34 total 50.03 last 76.25)
 - open/close 4212 (time 1775781187.35 total 60.04 last 70.85)
total: 4500 open/close in 63.67 seconds: 70.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1101 (time 1775781205.48 total 10.00 last 110.08)
 - /unlink 2175 (time 1775781215.49 total 20.01 last 107.32)
 - /unlink 3277 (time 1775781225.49 total 30.01 last 110.14)
total: 4500 /unlink in 39.47 seconds: 114.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 654 (time 1775781251.17 total 10.01 last 65.32)
 - open/close 1367 (time 1775781261.18 total 20.02 last 71.22)
 - open/close 2018 (time 1775781271.19 total 30.03 last 65.06)
 - open/close 2646 (time 1775781281.20 total 40.04 last 62.74)
 - open/close 3363 (time 1775781291.20 total 50.04 last 71.67)
 - open/close 3992 (time 1775781301.23 total 60.07 last 62.74)
total: 4500 open/close in 67.48 seconds: 66.69 ops/second
 - /unlink 1268 (time 1775781323.93 total 10.00 last 126.77)
 - /unlink 2554 (time 1775781333.94 total 20.01 last 128.46)
 - /unlink 3649 (time 1775781343.94 total 30.02 last 109.47)
total: 4500 /unlink in 36.74 seconds: 122.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 628 (time 1775781366.90 total 10.01 last 62.73)
 - open/close 1256 (time 1775781376.91 total 20.01 last 62.79)
 - open/close 1905 (time 1775781386.91 total 30.02 last 64.87)
 - open/close 2536 (time 1775781396.94 total 40.05 last 62.92)
 - open/close 3183 (time 1775781406.96 total 50.06 last 64.59)
 - open/close 3884 (time 1775781416.96 total 60.07 last 70.05)
total: 4500 open/close in 68.13 seconds: 66.05 ops/second
 - /unlink 1320 (time 1775781440.68 total 10.00 last 132.00)
 - /unlink 2732 (time 1775781450.69 total 20.01 last 141.12)
 - /unlink 3912 (time 1775781460.70 total 30.01 last 117.92)
total: 4500 /unlink in 34.71 seconds: 129.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 652 (time 1775781481.52 total 10.00 last 65.19)
 - open/close 1266 (time 1775781491.53 total 20.01 last 61.33)
 - open/close 2060 (time 1775781501.54 total 30.02 last 79.33)
 - open/close 2749 (time 1775781511.55 total 40.03 last 68.82)
 - open/close 3360 (time 1775781521.57 total 50.06 last 60.95)
 - open/close 3955 (time 1775781531.58 total 60.07 last 59.44)
total: 4500 open/close in 68.87 seconds: 65.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1333 (time 1775781556.14 total 10.00 last 133.26)
 - /unlink 2687 (time 1775781566.15 total 20.00 last 135.38)
 - /unlink 3888 (time 1775781576.15 total 30.01 last 120.08)
total: 4500 /unlink in 34.84 seconds: 129.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 532 (time 1775781597.22 total 10.00 last 53.18)
 - open/close 1048 (time 1775781607.23 total 20.01 last 51.55)
 - open/close 1721 (time 1775781617.23 total 30.01 last 67.30)
 - open/close 2476 (time 1775781627.24 total 40.03 last 75.38)
 - open/close 3132 (time 1775781637.25 total 50.04 last 65.55)
 - open/close 3883 (time 1775781647.26 total 60.04 last 75.06)
total: 4500 open/close in 68.25 seconds: 65.93 ops/second
 - /unlink 1377 (time 1775781670.68 total 10.01 last 137.61)
 - /unlink 2702 (time 1775781680.68 total 20.01 last 132.45)
 - /unlink 3897 (time 1775781690.68 total 30.01 last 119.49)
total: 4500 /unlink in 34.73 seconds: 129.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 624 (time 1775781710.87 total 10.01 last 62.35)
 - open/close 1218 (time 1775781720.88 total 20.01 last 59.39)
 - open/close 1880 (time 1775781730.89 total 30.02 last 66.11)
 - open/close 2638 (time 1775781740.89 total 40.03 last 75.77)
 - open/close 3443 (time 1775781750.92 total 50.06 last 80.28)
 - open/close 4083 (time 1775781760.94 total 60.07 last 63.90)
total: 4500 open/close in 66.47 seconds: 67.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1257 (time 1775781782.81 total 10.00 last 125.64)
 - /unlink 2289 (time 1775781792.83 total 20.02 last 103.07)
 - /unlink 3401 (time 1775781802.83 total 30.02 last 111.14)
 - /unlink 4494 (time 1775781812.86 total 40.05 last 109.02)
total: 4500 /unlink in 40.10 seconds: 112.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 648 (time 1775781830.28 total 10.01 last 64.76)
 - open/close 1208 (time 1775781840.30 total 20.02 last 55.91)
 - open/close 1792 (time 1775781850.31 total 30.03 last 58.34)
 - open/close 2498 (time 1775781860.32 total 40.04 last 70.55)
 - open/close 3193 (time 1775781870.32 total 50.05 last 69.46)
 - open/close 3890 (time 1775781880.32 total 60.05 last 69.70)
total: 4500 open/close in 68.77 seconds: 65.43 ops/second
 - /unlink 1450 (time 1775781903.82 total 10.00 last 144.97)
 - /unlink 2635 (time 1775781913.82 total 20.00 last 118.49)
 - /unlink 4012 (time 1775781923.83 total 30.01 last 137.64)
total: 4500 /unlink in 34.19 seconds: 131.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 767 (time 1775781944.76 total 10.03 last 76.50)
 - open/close 1584 (time 1775781954.77 total 20.04 last 81.60)
 - open/close 2286 (time 1775781964.77 total 30.04 last 70.18)
 - open/close 2982 (time 1775781974.78 total 40.05 last 69.56)
 - open/close 3718 (time 1775781984.79 total 50.06 last 73.51)
 - open/close 4341 (time 1775781994.79 total 60.06 last 62.29)
total: 4500 open/close in 62.97 seconds: 71.46 ops/second
 - /unlink 1240 (time 1775782012.67 total 10.00 last 123.96)
 - /unlink 2642 (time 1775782022.68 total 20.01 last 140.12)
 - /unlink 3915 (time 1775782032.68 total 30.01 last 127.27)
total: 4500 /unlink in 34.37 seconds: 130.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 537 (time 1775782055.20 total 10.01 last 53.64)
 - open/close 1153 (time 1775782065.20 total 20.02 last 61.56)
 - open/close 1818 (time 1775782075.21 total 30.02 last 66.48)
 - open/close 2535 (time 1775782085.21 total 40.02 last 71.69)
 - open/close 3401 (time 1775782095.22 total 50.03 last 86.51)
 - open/close 4140 (time 1775782105.22 total 60.04 last 73.87)
total: 4500 open/close in 65.07 seconds: 69.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1054 (time 1775782125.92 total 10.01 last 105.27)
 - /unlink 2101 (time 1775782135.92 total 20.02 last 104.64)
 - /unlink 3346 (time 1775782145.92 total 30.02 last 124.50)
total: 4500 /unlink in 37.96 seconds: 118.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 675 (time 1775782169.48 total 10.02 last 67.35)
 - open/close 1382 (time 1775782179.48 total 20.02 last 70.69)
 - open/close 2054 (time 1775782189.49 total 30.03 last 67.19)
 - open/close 2734 (time 1775782199.49 total 40.03 last 67.97)
 - open/close 3358 (time 1775782209.50 total 50.04 last 62.32)
 - open/close 4025 (time 1775782219.51 total 60.05 last 66.67)
total: 4500 open/close in 67.83 seconds: 66.34 ops/second
 - /unlink 1240 (time 1775782241.95 total 10.01 last 123.86)
 - /unlink 2507 (time 1775782251.95 total 20.01 last 126.68)
 - /unlink 3836 (time 1775782261.96 total 30.02 last 132.78)
total: 4500 /unlink in 35.38 seconds: 127.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 749 (time 1775782282.78 total 10.01 last 74.84)
 - open/close 1448 (time 1775782292.79 total 20.01 last 69.88)
 - open/close 1952 (time 1775782302.81 total 30.03 last 50.29)
 - open/close 2487 (time 1775782312.83 total 40.05 last 53.40)
 - open/close 3219 (time 1775782322.83 total 50.05 last 73.20)
 - open/close 3997 (time 1775782332.85 total 60.08 last 77.59)
total: 4500 open/close in 67.37 seconds: 66.79 ops/second
 - /unlink 1207 (time 1775782354.30 total 10.00 last 120.65)
 - /unlink 2436 (time 1775782364.32 total 20.02 last 122.67)
 - /unlink 3632 (time 1775782374.33 total 30.03 last 119.46)
total: 4500 /unlink in 37.59 seconds: 119.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 557 (time 1775782398.43 total 10.00 last 55.69)
 - open/close 1179 (time 1775782408.44 total 20.01 last 62.14)
 - open/close 1921 (time 1775782418.47 total 30.04 last 74.00)
 - open/close 2611 (time 1775782428.47 total 40.04 last 68.99)
 - open/close 3345 (time 1775782438.47 total 50.04 last 73.39)
 - open/close 4043 (time 1775782448.48 total 60.05 last 69.74)
total: 4500 open/close in 66.24 seconds: 67.93 ops/second
 - /unlink 1415 (time 1775782469.85 total 10.00 last 141.48)
 - /unlink 2692 (time 1775782479.85 total 20.00 last 127.67)
 - /unlink 3807 (time 1775782489.86 total 30.01 last 111.47)
total: 4500 /unlink in 34.57 seconds: 130.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 655 (time 1775782509.79 total 10.01 last 65.44)
 - open/close 1273 (time 1775782519.80 total 20.01 last 61.79)
 - open/close 1949 (time 1775782529.81 total 30.02 last 67.52)
 - open/close 2590 (time 1775782539.82 total 40.04 last 64.01)
 - open/close 3209 (time 1775782549.83 total 50.04 last 61.87)
 - open/close 3658 (time 1775782559.85 total 60.07 last 44.79)
 - open/close 4212 (time 1775782569.86 total 70.07 last 55.35)
total: 4500 open/close in 73.74 seconds: 61.02 ops/second
 - /unlink 1361 (time 1775782588.79 total 10.00 last 136.08)
 - /unlink 2759 (time 1775782598.80 total 20.01 last 139.67)
 - /unlink 4129 (time 1775782608.81 total 30.02 last 136.93)
total: 4500 /unlink in 32.87 seconds: 136.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 716 (time 1775782626.28 total 10.01 last 71.53)
 - open/close 1470 (time 1775782636.28 total 20.01 last 75.37)
 - open/close 2221 (time 1775782646.28 total 30.02 last 75.08)
 - open/close 2964 (time 1775782656.30 total 40.03 last 74.22)
 - open/close 3627 (time 1775782666.30 total 50.03 last 66.25)
 - open/close 4344 (time 1775782676.35 total 60.08 last 71.38)
total: 4500 open/close in 62.62 seconds: 71.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1288 (time 1775782692.70 total 10.00 last 128.76)
 - /unlink 2530 (time 1775782702.70 total 20.00 last 124.20)
 - /unlink 3816 (time 1775782712.70 total 30.01 last 128.55)
total: 4500 /unlink in 34.80 seconds: 129.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 648 (time 1775782732.71 total 10.01 last 64.76)
 - open/close 1316 (time 1775782742.72 total 20.02 last 66.72)
 - open/close 2029 (time 1775782752.72 total 30.02 last 71.30)
 - open/close 2690 (time 1775782762.73 total 40.03 last 66.06)
 - open/close 3500 (time 1775782772.73 total 50.03 last 80.97)
 - open/close 4207 (time 1775782782.75 total 60.05 last 70.56)
total: 4500 open/close in 64.70 seconds: 69.55 ops/second
 - /unlink 1144 (time 1775782803.67 total 10.00 last 114.40)
 - /unlink 2679 (time 1775782813.68 total 20.01 last 153.36)
 - /unlink 4101 (time 1775782823.68 total 30.02 last 142.11)
total: 4500 /unlink in 33.08 seconds: 136.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 744 (time 1775782841.64 total 10.00 last 74.37)
 - open/close 1450 (time 1775782851.65 total 20.01 last 70.55)
 - open/close 2096 (time 1775782861.67 total 30.03 last 64.47)
 - open/close 2746 (time 1775782871.67 total 40.03 last 64.99)
 - open/close 3351 (time 1775782881.68 total 50.04 last 60.44)
 - open/close 4084 (time 1775782891.68 total 60.05 last 73.26)
total: 4500 open/close in 65.42 seconds: 68.78 ops/second
 - /unlink 1204 (time 1775782912.17 total 10.00 last 120.36)
 - /unlink 2592 (time 1775782922.18 total 20.02 last 138.63)
 - /unlink 3859 (time 1775782932.19 total 30.02 last 126.59)
total: 4500 /unlink in 35.07 seconds: 128.31 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11184: 94168 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg112-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg112-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server
stop mds service on oleg112-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server
stop mds service on oleg112-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server

unloading modules via unload_modules_local on: 'oleg112-server'
oleg112-server: oleg112-server.virtnet: executing unload_modules_local
oleg112-server: 
modules unloaded.