|
2017-03-24
§
|
| 02:26 |
<Krinkle> |
Reminder to incident doc writer: Logstash was (and is) not responsive serving Kibana-rendered errors about logstash Service unavailable |
[production] |
| 02:25 |
<Krinkle> |
All apaches are back up |
[production] |
| 02:24 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.17/extensions/Wikidata: revert (duration: 02m 34s) |
[production] |
| 02:24 |
<MaxSem> |
Killed l10nupdate on tin, was blocking emergency pushes |
[production] |
| 02:22 |
<Krinkle> |
Hard-killed all l10nupdate processes and rm'ed scap lock |
[production] |
| 02:11 |
<Krinkle> |
Removing xhgui.results entries from before 1 December 2016 in MongoDB on tungsten (T161196) |
[production] |
| 01:45 |
<mutante> |
bacula - on helium, attempt to start bacula-director process, attempt to fix permissions on key files as codified in director.pp |
[production] |
| 01:40 |
<catrope@tin> |
Finished scap: Wikidata cherry-picks (with i18n) (duration: 25m 03s) |
[production] |
| 01:15 |
<catrope@tin> |
Started scap: Wikidata cherry-picks (with i18n) |
[production] |
|
2017-03-23
§
|
| 23:26 |
<Krinkle> |
Removing xhgui.results entries from before 1 June 2016 (T161196) |
[production] |
| 23:12 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.17/extensions/ORES: SWAT: [[gerrit:344555|Stats: Invert "false" thresholds so they are correct]] T161250 (duration: 00m 52s) |
[production] |
| 23:05 |
<Pchelolo> |
update RESTBase to 2536b25c7 - eqiad |
[production] |
| 22:56 |
<Pchelolo> |
update RESTBase to 2536b25c7 - staging |
[production] |
| 22:39 |
<Pchelolo> |
update RESTBase to 2536b25c7 - codfw |
[production] |
| 21:36 |
<krinkle@tin> |
Synchronized wmf-config/StartProfiler.php: (no justification provided) (duration: 00m 53s) |
[production] |
| 21:05 |
<ejegg> |
rolled back payments-wiki to 9622a4bf331426ce85362e2fdb442f226068e971 |
[production] |
| 21:00 |
<ejegg> |
updated payments from 9622a4bf331426ce85362e2fdb442f226068e971 to bb956bf1ea7f0c7a9b08877cce7f25b201bd2bbf |
[production] |
| 20:38 |
<andrewbogott> |
migrating tools-exec-1401 to labvirt1001 |
[tools] |
| 20:17 |
<ottomata> |
moved all analytics cluster cron jobs (camus and other) from analytics1027 to analytics1003: T159527 |
[analytics] |
| 20:16 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17 |
[production] |
| 20:14 |
<ottomata> |
earlier today: upgraded from cdh5.2 to cdh5.10 on analytics1030, somehow we missed it! :o |
[analytics] |
| 19:56 |
<andrewbogott> |
migrating tools-exec-1408 to labvirt1001 |
[tools] |
| 19:34 |
<thcipriani@tin> |
Synchronized php: Swap symlink for 1.29.0-wmf.17 (duration: 00m 43s) |
[production] |
| 19:11 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.17 |
[production] |
| 19:01 |
<andrewbogott> |
migrating tools-exec-1407 to labvirt1001 |
[tools] |
| 18:51 |
<bblack> |
systemctl enable+start of lldpd on cp2009, cp1051, cp1061 (mysteriously dead and disabled) |
[production] |
| 18:16 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove exception on Other Projects sidebar for Dutch Wikipedia (T159634) (duration: 00m 47s) |
[production] |
| 18:04 |
<Pchelolo> |
update RESTBase to 9d2b393fb - production |
[production] |
| 17:52 |
<Pchelolo> |
update RESTBase to 9d2b393fb - staging |
[production] |
| 16:46 |
<mobrovac@tin> |
Started restart [parsoid/deploy@0c22f72]: (no justification provided) |
[production] |
| 16:45 |
<_joe_> |
reenabling puppet on all jobqueue redises |
[production] |
| 16:37 |
<andrewbogott> |
migrating tools-webgrid-lighttpd-1402 and 1407 to labvirt1001 (testing labvirt1001 and easing CPU load on labvirt1010) |
[tools] |
| 16:24 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 16:22 |
<hashar> |
Merged operations/puppet.git Jenkins job in a single one that runs tox then rake - T160923 |
[production] |
| 16:10 |
<urandom> |
T111113: Live-hacking client encryption to be non-optional, to verify cqlsh encryption, restbase1007-a.eqiad.wmnet |
[production] |
| 16:07 |
<mobrovac> |
restbase deploy 752ca4b7 |
[production] |
| 16:07 |
<mobrovac> |
restbase deploying 752ca4b7 |
[releng] |
| 15:52 |
<hashar> |
Deleting integration-slave-trusty-1011 m1.large. One less perm slave to take care about |
[releng] |
| 15:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 15:32 |
<moritzm> |
upgrading restbase-test* to Linux 4.9 |
[production] |
| 14:59 |
<akosiaris> |
enabling and running puppet on rdb200X fleet in a rolling restart scheme |
[production] |
| 14:59 |
<akosiaris> |
disabled puppet on rdb* fleet |
[production] |
| 14:56 |
<andrewbogott> |
dist-upgrading labvirt1001 and rebooting it a few times |
[production] |
| 14:22 |
<moritzm> |
installing exim4 updates from jessie point release |
[production] |
| 14:03 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T137191 (duration: 00m 48s) |
[production] |
| 14:02 |
<hashar> |
deployment-ms-be01 and deployment-ms-be02 : Lower Swift replicator on, upgrade package, reboot hosts. T160990 |
[releng] |
| 13:53 |
<dcausse> |
cirrus: refreshing comp suggest indices in elastic@eqiad to measure times |
[production] |
| 12:59 |
<marostegui> |
Deploy schema change s4 on db1064 https://phabricator.wikimedia.org/T160415 - https://phabricator.wikimedia.org/T73563 |
[production] |
| 12:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1059, depool db1064 T160415 - T73563 (duration: 00m 43s) |
[production] |
| 12:33 |
<Sebastian-WMSE> |
Deploy latest from Git master: e41c0c2 (T146459, T158954) |
[wikispeech] |