== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:36:05 (1788586565) Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg247-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: kmemleak disabled oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg247-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg247-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 768 (time 1788586779.82 total 10.01 last 76.74) - open/close 1545 (time 1788586789.83 total 20.01 last 77.65) - open/close 2390 (time 1788586799.84 total 30.02 last 84.45) - open/close 3089 (time 1788586809.84 total 40.02 last 69.88) - open/close 3791 (time 1788586819.85 total 50.03 last 70.15) - open/close 4384 (time 1788586829.85 total 60.03 last 59.30) total: 4500 open/close in 61.76 seconds: 72.87 ops/second - /unlink 1339 (time 1788586846.70 total 10.00 last 133.88) - /unlink 2612 (time 1788586856.70 total 20.00 last 127.28) - /unlink 4000 (time 1788586866.70 total 30.00 last 138.79) total: 4500 /unlink in 33.42 seconds: 134.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 697 (time 1788586885.86 total 10.01 last 69.65) - open/close 1466 (time 1788586895.89 total 20.03 last 76.69) - open/close 2192 (time 1788586905.89 total 30.04 last 72.59) - open/close 2955 (time 1788586915.91 total 40.05 last 76.19) - open/close 3669 (time 1788586925.91 total 50.06 last 71.34) - open/close 4344 (time 1788586935.92 total 60.06 last 67.47) total: 4500 open/close in 62.36 seconds: 72.16 ops/second - /unlink 1253 (time 1788586953.45 total 10.00 last 125.27) - /unlink 2674 (time 1788586963.45 total 20.00 last 142.08) - /unlink 4078 (time 1788586973.46 total 30.01 last 140.35) total: 4500 /unlink in 33.10 seconds: 135.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 734 (time 1788586992.33 total 10.03 last 73.21) - open/close 1489 (time 1788587002.34 total 20.04 last 75.43) - open/close 2239 (time 1788587012.34 total 30.04 last 74.97) - open/close 2994 (time 1788587022.36 total 40.06 last 75.38) - open/close 3706 (time 1788587032.36 total 50.06 last 71.16) - open/close 4451 (time 1788587042.37 total 60.07 last 74.43) total: 4500 open/close in 60.76 seconds: 74.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1262 (time 1788587059.09 total 10.00 last 126.16) - /unlink 2497 (time 1788587069.09 total 20.00 last 123.49) - /unlink 3894 (time 1788587079.09 total 30.01 last 139.66) total: 4500 /unlink in 34.38 seconds: 130.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 639 (time 1788587099.14 total 10.00 last 63.87) - open/close 1321 (time 1788587109.16 total 20.03 last 68.06) - open/close 2031 (time 1788587119.17 total 30.03 last 70.95) - open/close 2703 (time 1788587129.18 total 40.05 last 67.11) - open/close 3420 (time 1788587139.22 total 50.08 last 71.43) - open/close 4177 (time 1788587149.23 total 60.10 last 75.58) total: 4500 open/close in 64.90 seconds: 69.34 ops/second - /unlink 1333 (time 1788587168.96 total 10.00 last 133.24) - /unlink 2574 (time 1788587178.96 total 20.01 last 124.06) - /unlink 3887 (time 1788587188.96 total 30.01 last 131.26) total: 4500 /unlink in 35.20 seconds: 127.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 648 (time 1788587209.23 total 10.01 last 64.75) - open/close 1158 (time 1788587219.25 total 20.02 last 50.95) - open/close 1777 (time 1788587229.25 total 30.03 last 61.84) - open/close 2503 (time 1788587239.27 total 40.04 last 72.52) - open/close 3102 (time 1788587249.28 total 50.05 last 59.84) - open/close 3754 (time 1788587259.30 total 60.07 last 65.04) total: 4500 open/close in 69.99 seconds: 64.29 ops/second - /unlink 1514 (time 1788587282.92 total 10.00 last 151.36) - /unlink 2933 (time 1788587292.92 total 20.00 last 141.89) - /unlink 4199 (time 1788587302.92 total 30.01 last 126.50) total: 4500 /unlink in 32.69 seconds: 137.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 618 (time 1788587321.35 total 10.00 last 61.80) - open/close 1347 (time 1788587331.36 total 20.01 last 72.81) - open/close 2059 (time 1788587341.37 total 30.02 last 71.13) - open/close 2837 (time 1788587351.38 total 40.03 last 77.74) - open/close 3618 (time 1788587361.38 total 50.04 last 78.06) - open/close 4356 (time 1788587371.39 total 60.04 last 73.76) total: 4500 open/close in 61.82 seconds: 72.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1280 (time 1788587388.10 total 10.00 last 127.99) - /unlink 2683 (time 1788587398.10 total 20.00 last 140.26) - /unlink 4057 (time 1788587408.10 total 30.01 last 137.35) total: 4500 /unlink in 34.43 seconds: 130.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 559 (time 1788587428.29 total 10.02 last 55.79) - open/close 1252 (time 1788587438.31 total 20.04 last 69.16) - open/close 1997 (time 1788587448.32 total 30.05 last 74.46) - open/close 2633 (time 1788587458.32 total 40.05 last 63.59) - open/close 3414 (time 1788587468.32 total 50.05 last 78.10) - open/close 4191 (time 1788587478.32 total 60.05 last 77.68) total: 4500 open/close in 64.20 seconds: 70.10 ops/second - /unlink 1384 (time 1788587497.49 total 10.00 last 138.38) - /unlink 2725 (time 1788587507.50 total 20.01 last 133.98) - /unlink 3930 (time 1788587517.50 total 30.01 last 120.47) total: 4500 /unlink in 33.87 seconds: 132.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 730 (time 1788587536.65 total 10.07 last 72.51) - open/close 1461 (time 1788587546.65 total 20.07 last 73.06) - open/close 2144 (time 1788587556.65 total 30.07 last 68.29) - open/close 2820 (time 1788587566.66 total 40.08 last 67.58) - open/close 3537 (time 1788587576.66 total 50.08 last 71.65) - open/close 4257 (time 1788587586.68 total 60.10 last 71.92) total: 4500 open/close in 63.63 seconds: 70.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1271 (time 1788587605.55 total 10.00 last 127.07) - /unlink 2553 (time 1788587615.56 total 20.01 last 128.14) - /unlink 3829 (time 1788587625.56 total 30.01 last 127.52) total: 4500 /unlink in 35.98 seconds: 125.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 599 (time 1788587647.43 total 10.02 last 59.76) - open/close 1233 (time 1788587657.43 total 20.03 last 63.38) - open/close 1966 (time 1788587667.45 total 30.04 last 73.21) - open/close 2619 (time 1788587677.46 total 40.05 last 65.22) - open/close 3246 (time 1788587687.47 total 50.06 last 62.65) - open/close 3898 (time 1788587697.47 total 60.07 last 65.16) total: 4500 open/close in 68.67 seconds: 65.53 ops/second - /unlink 1389 (time 1788587720.55 total 10.00 last 138.84) - /unlink 2576 (time 1788587730.56 total 20.01 last 118.60) - /unlink 3923 (time 1788587740.56 total 30.02 last 134.64) total: 4500 /unlink in 34.40 seconds: 130.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 715 (time 1788587760.43 total 10.01 last 71.46) - open/close 1386 (time 1788587770.44 total 20.02 last 67.03) - open/close 2040 (time 1788587780.44 total 30.02 last 65.36) - open/close 2716 (time 1788587790.45 total 40.03 last 67.57) - open/close 3418 (time 1788587800.46 total 50.04 last 70.12) - open/close 4144 (time 1788587810.46 total 60.04 last 72.58) total: 4500 open/close in 64.59 seconds: 69.67 ops/second - /unlink 1463 (time 1788587829.73 total 10.01 last 146.21) - /unlink 2811 (time 1788587839.73 total 20.01 last 134.77) - /unlink 4285 (time 1788587849.74 total 30.01 last 147.30) total: 4500 /unlink in 31.55 seconds: 142.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 659 (time 1788587866.24 total 10.00 last 65.88) - open/close 1276 (time 1788587876.26 total 20.01 last 61.63) - open/close 2001 (time 1788587886.26 total 30.02 last 72.45) - open/close 2645 (time 1788587896.27 total 40.03 last 64.37) - open/close 3387 (time 1788587906.28 total 50.03 last 74.14) - open/close 3939 (time 1788587916.29 total 60.04 last 55.14) total: 4500 open/close in 69.15 seconds: 65.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1293 (time 1788587940.83 total 10.00 last 129.27) - /unlink 2575 (time 1788587950.84 total 20.01 last 128.12) - /unlink 3963 (time 1788587960.85 total 30.01 last 138.71) total: 4500 /unlink in 34.19 seconds: 131.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 767 (time 1788587980.07 total 10.01 last 76.65) - open/close 1425 (time 1788587990.07 total 20.01 last 65.76) - open/close 2079 (time 1788588000.08 total 30.01 last 65.39) - open/close 2833 (time 1788588010.09 total 40.03 last 75.28) - open/close 3497 (time 1788588020.09 total 50.03 last 66.39) - open/close 4283 (time 1788588030.10 total 60.04 last 78.55) total: 4500 open/close in 63.42 seconds: 70.96 ops/second - /unlink 1179 (time 1788588048.53 total 10.00 last 117.88) - /unlink 2314 (time 1788588058.54 total 20.01 last 113.36) - /unlink 3387 (time 1788588068.54 total 30.01 last 107.29) total: 4500 /unlink in 39.70 seconds: 113.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 619 (time 1788588093.47 total 10.00 last 61.90) - open/close 1328 (time 1788588103.48 total 20.01 last 70.85) - open/close 2085 (time 1788588113.49 total 30.02 last 75.61) - open/close 2790 (time 1788588123.55 total 40.07 last 70.13) - open/close 3440 (time 1788588133.56 total 50.09 last 64.91) - open/close 4069 (time 1788588143.57 total 60.09 last 62.87) total: 4500 open/close in 66.63 seconds: 67.54 ops/second - /unlink 1051 (time 1788588166.18 total 10.00 last 105.10) - /unlink 2211 (time 1788588176.18 total 20.01 last 115.92) - /unlink 3379 (time 1788588186.19 total 30.01 last 116.77) total: 4500 /unlink in 38.84 seconds: 115.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 691 (time 1788588211.28 total 10.01 last 69.03) - open/close 1341 (time 1788588221.29 total 20.02 last 64.93) - open/close 2022 (time 1788588231.29 total 30.02 last 68.09) - open/close 2702 (time 1788588241.29 total 40.02 last 68.00) - open/close 3414 (time 1788588251.30 total 50.03 last 71.11) - open/close 4064 (time 1788588261.30 total 60.04 last 64.98) total: 4500 open/close in 66.56 seconds: 67.61 ops/second - /unlink 1488 (time 1788588282.12 total 10.01 last 148.67) - /unlink 2732 (time 1788588292.12 total 20.01 last 124.40) - /unlink 3857 (time 1788588302.12 total 30.01 last 112.47) total: 4500 /unlink in 34.46 seconds: 130.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 690 (time 1788588321.65 total 10.01 last 68.91) - open/close 1432 (time 1788588331.66 total 20.02 last 74.18) - open/close 2180 (time 1788588341.67 total 30.02 last 74.74) - open/close 2908 (time 1788588351.67 total 40.03 last 72.77) - open/close 3571 (time 1788588361.69 total 50.04 last 66.19) - open/close 4187 (time 1788588371.71 total 60.06 last 61.48) total: 4500 open/close in 64.70 seconds: 69.55 ops/second - /unlink 1295 (time 1788588391.22 total 10.00 last 129.45) - /unlink 2744 (time 1788588401.22 total 20.01 last 144.87) - /unlink 3938 (time 1788588411.22 total 30.01 last 119.38) total: 4500 /unlink in 34.81 seconds: 129.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 657 (time 1788588431.96 total 10.01 last 65.62) - open/close 1367 (time 1788588441.97 total 20.01 last 70.98) - open/close 2109 (time 1788588451.98 total 30.03 last 74.12) - open/close 2732 (time 1788588461.99 total 40.03 last 62.25) - open/close 3401 (time 1788588471.99 total 50.04 last 66.87) - open/close 3969 (time 1788588481.99 total 60.04 last 56.80) total: 4500 open/close in 69.25 seconds: 64.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1530 (time 1788588506.05 total 10.01 last 152.81) - /unlink 2764 (time 1788588516.05 total 20.01 last 123.39) - /unlink 4213 (time 1788588526.06 total 30.03 last 144.71) total: 4500 /unlink in 32.15 seconds: 139.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 602 (time 1788588543.93 total 10.01 last 60.17) - open/close 1283 (time 1788588553.94 total 20.01 last 68.04) - open/close 1988 (time 1788588563.95 total 30.02 last 70.45) - open/close 2666 (time 1788588573.95 total 40.03 last 67.75) - open/close 3294 (time 1788588583.97 total 50.05 last 62.69) - open/close 4053 (time 1788588593.97 total 60.05 last 75.87) total: 4500 open/close in 65.91 seconds: 68.28 ops/second - /unlink 1445 (time 1788588614.23 total 10.01 last 144.41) - /unlink 2775 (time 1788588624.24 total 20.02 last 132.83) - /unlink 4108 (time 1788588634.24 total 30.02 last 133.25) total: 4500 /unlink in 33.69 seconds: 133.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 577 (time 1788588653.32 total 10.02 last 57.57) - open/close 1219 (time 1788588663.32 total 20.02 last 64.19) - open/close 1901 (time 1788588673.32 total 30.03 last 68.18) - open/close 2603 (time 1788588683.34 total 40.05 last 70.07) - open/close 3137 (time 1788588693.36 total 50.07 last 53.29) - open/close 3651 (time 1788588703.37 total 60.07 last 51.39) - open/close 4340 (time 1788588713.37 total 70.07 last 68.88) total: 4500 open/close in 72.37 seconds: 62.18 ops/second - /unlink 1192 (time 1788588731.25 total 10.01 last 119.13) - /unlink 2517 (time 1788588741.26 total 20.01 last 132.45) - /unlink 3874 (time 1788588751.26 total 30.01 last 135.68) total: 4500 /unlink in 34.82 seconds: 129.22 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95388 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: kmemleak disabled modules unloaded.