1201-1250 of 10000 results (77ms)
2019-10-07 §
10:55 <mobrovac@deploy1001> Started deploy [restbase/deploy@5321aac]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #5 [production]
10:54 <mobrovac@deploy1001> Finished deploy [restbase/deploy@5321aac]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #4 (duration: 04m 27s) [production]
10:50 <mobrovac@deploy1001> Started deploy [restbase/deploy@5321aac]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #4 [production]
10:48 <mobrovac@deploy1001> Finished deploy [restbase/deploy@5321aac]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #3 (duration: 03m 53s) [production]
10:44 <mobrovac@deploy1001> Started deploy [restbase/deploy@5321aac]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #3 [production]
10:37 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:541217| Bumping portals to master (T128546)]] (duration: 00m 51s) [production]
10:36 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:541217| Bumping portals to master (T128546)]] (duration: 00m 53s) [production]
10:31 <_joe_> uploading confd 0.16.0 to stretch [production]
10:21 <mobrovac@deploy1001> Finished deploy [restbase/deploy@1798e39]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #2 (duration: 01m 56s) [production]
10:19 <mobrovac@deploy1001> Started deploy [restbase/deploy@1798e39]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests, take #2 [production]
10:16 <mobrovac@deploy1001> Finished deploy [restbase/deploy@1798e39]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests - T233127 T234772 (duration: 05m 58s) [production]
10:10 <mobrovac@deploy1001> Started deploy [restbase/deploy@1798e39]: Skip checking resources on start-up, add banwiki, add metrics/mediarequests end points and log all VE requests - T233127 T234772 [production]
09:55 <marostegui> Deploy schema change on db2129 (s6 codfw master), this will generate lag on s6 codfw - T233135 T234066 [production]
08:34 <hashar> gerrit: force reindexing all changes ( gerrit index start changes --force ) [production]
08:33 <hashar> gerrit: `gerrit index start changes --force` # T233989 [releng]
08:32 <hashar> gerrit: `gerrit index start changes` # T233989 [releng]
07:09 <marostegui> Remove grants for dbproxy1006 on m1 databases - T231280 [production]
06:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3315 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9246 and previous config saved to /var/cache/conftool/dbconfig/20191007-065645-marostegui.json [production]
06:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es1011 T227138 (duration: 01m 10s) [production]
06:08 <elukey> upgrade python-kafka on eventlog1002 to 1.4.7-1 (manually via dpkg -i) [analytics]
06:08 <elukey> upgrade python-kafka on eventlog1002 to 1.4.7-1 (manually via dpkg -i) - T222941 [production]
05:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
05:45 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
05:25 <marostegui> Deploy schema change on db2124 T233135 T234066 [production]
05:10 <marostegui> The above was for db2095:3316 T234704 [production]
05:08 <marostegui> Stop replication on db2076 to modify triggers on db2096:3316 T234704 [production]
05:02 <marostegui> Fix replication on labsdb1011:s8 [production]
04:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3316 for schema change T233135 T234066', diff saved to https://phabricator.wikimedia.org/P9245 and previous config saved to /var/cache/conftool/dbconfig/20191007-045411-marostegui.json [production]
2019-10-06 §
20:11 <Urbanecm> mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Racconish /home/urbanecm/T234741 (T234741) [production]
19:15 <marostegui> Reload haproxy on dbproxy1010, dbproxy1011, dbproxy1018, dbproxy1019 [production]
08:59 <wm-bot> <lucaswerkmeister> deployed 6df8be4176 (settings page improvements) [tools.speedpatrolling]
07:55 <wm-bot> <lucaswerkmeister> deployed temporary debug code [tools.speedpatrolling]
07:51 <wm-bot> <lucaswerkmeister> deployed 4dc37fbd57 (helpful info for not-fully-privileged users) [tools.speedpatrolling]
06:47 <elukey> delete old cron entry 'xenon_generate_svgs' (user xenon) on webperf[12]002 to reduce cronspam [production]
2019-10-05 §
18:18 <elukey> kill/restart mediawiki-history-reduced oozie coord to pick up the new druid_loader.py version on HDFS [analytics]
06:49 <elukey> force umount/remount of /mnt/hdfs on an-coord1001 - processes stuck in D state, fuser proc consuming a ton of memory [analytics]
06:48 <elukey> force umount/remount of /mnt/hdfs on an-coord1001 - processes stuck in D state, fuser proc consuming a ton of memory [production]
2019-10-04 §
22:06 <mutante> ms-be1020 - power cycle via mgmt - host down [production]
21:43 <bd808> `sudo exec-manage repool tools-sgeexec-0923.tools.eqiad.wmflabs` [tools]
21:26 <bd808> Rebooting tools-sgeexec-0923 after lots of messing about with a broken update-initramfs build [tools]
21:13 <hauskater> gerrit: Ran replication start mediawiki/extensions/WikiSEO --wait # New repository [releng]
20:43 <krinkle@deploy1001> Synchronized w/static.php: 9648e03, 97d9384 (duration: 00m 53s) [production]
20:41 <mutante> deploy1001 / deploy2001 - remove python-pygerrit2 (version for python3 is needed instead) [production]
20:35 <bd808> Manually running `/usr/bin/python3 /usr/bin/unattended-upgrade` on tools-sgeexec-0923 [tools]
20:33 <bd808> Killed 2 /usr/bin/unattended-upgrade procs on tools-sgeexec-0923 that seemed stuck [tools]
20:32 <mutante> gerrit1001 - scp /usr/share/java/mysql-connector-java.jar from cobalt into /usr/share/java/ on gerrit1001 and then symlink into /var/lib/gerrit2/review_site/lib/ (T222391) [production]
19:34 <marxarelli> re-running `mediawiki/tools/release/make-release/branch.py --bundle '*' ...` due to https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/540953 cc: T232024 [releng]
19:27 <mutante> wtp1025 - mediawiki appserver classes are being applied, install in progress will trigger some new icinga alerts [production]
19:17 <Krinkle> Fix Jenkins Console section "setup quibble prepare" pattern to account for recent changes on "mwext-php72-phan-docker" which was eating the pattern incorrectly [releng]
18:55 <marxarelli> branching REL1_34 for remaining extensions/skins and core, `mediawiki/tools/release/make-release/branch.py REL1_34 --bundle '*' --core --branchpoint master --core-version 1.34.0` (note: core branchpoint is overridden in settings.yaml at base of wmf/1.34.0-wmf.25) cc: T232024 [releng]