|
2017-03-24
§
|
| 06:10 |
<Krinkle> |
Removing xhgui.results entries before 1-Dec-2016 finished. Running xhgui->command(compact=>results) now. T161196 |
[production] |
| 02:31 |
<Krinkle> |
Reverted patch - https://gerrit.wikimedia.org/r/#/c/344569/ |
[production] |
| 02:30 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.17) (duration: 08m 35s) |
[production] |
| 02:28 |
<Krinkle> |
Reminder to incident doc writer: Was difficult figuring out what the last "real" patch was, scap message for SAL is manually written (not says which commit in which repo), and git log contains noise from security patches. We need simple revert options from the flat git tree at /srv/mediawiki |
[production] |
| 02:26 |
<Krinkle> |
Reminder to incident doc writer: Logstash was (and is) not responsive serving Kibana-rendered errors about logstash Service unavailable |
[production] |
| 02:25 |
<Krinkle> |
All apaches are back up |
[production] |
| 02:24 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.17/extensions/Wikidata: revert (duration: 02m 34s) |
[production] |
| 02:24 |
<MaxSem> |
Killed l10nupdate on tin, was blocking emergency pushes |
[production] |
| 02:22 |
<Krinkle> |
Hard-killed all l10nupdate processes and rm'ed scap lock |
[production] |
| 02:11 |
<Krinkle> |
Removing xhgui.results entries from before 1 December 2016 in MongoDB on tungsten (T161196) |
[production] |
| 01:45 |
<mutante> |
bacula - on helium, attempt to start bacula-director process, attempt to fix permissions on key files as codified in director.pp |
[production] |
| 01:40 |
<catrope@tin> |
Finished scap: Wikidata cherry-picks (with i18n) (duration: 25m 03s) |
[production] |
| 01:15 |
<catrope@tin> |
Started scap: Wikidata cherry-picks (with i18n) |
[production] |
|
2017-03-23
§
|
| 23:26 |
<Krinkle> |
Removing xhgui.results entries from before 1 June 2016 (T161196) |
[production] |
| 23:12 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.17/extensions/ORES: SWAT: [[gerrit:344555|Stats: Invert "false" thresholds so they are correct]] T161250 (duration: 00m 52s) |
[production] |
| 23:05 |
<Pchelolo> |
update RESTBase to 2536b25c7 - eqiad |
[production] |
| 22:56 |
<Pchelolo> |
update RESTBase to 2536b25c7 - staging |
[production] |
| 22:39 |
<Pchelolo> |
update RESTBase to 2536b25c7 - codfw |
[production] |
| 21:36 |
<krinkle@tin> |
Synchronized wmf-config/StartProfiler.php: (no justification provided) (duration: 00m 53s) |
[production] |
| 21:05 |
<ejegg> |
rolled back payments-wiki to 9622a4bf331426ce85362e2fdb442f226068e971 |
[production] |
| 21:00 |
<ejegg> |
updated payments from 9622a4bf331426ce85362e2fdb442f226068e971 to bb956bf1ea7f0c7a9b08877cce7f25b201bd2bbf |
[production] |
| 20:16 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17 |
[production] |
| 19:34 |
<thcipriani@tin> |
Synchronized php: Swap symlink for 1.29.0-wmf.17 (duration: 00m 43s) |
[production] |
| 19:11 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.17 |
[production] |
| 18:51 |
<bblack> |
systemctl enable+start of lldpd on cp2009, cp1051, cp1061 (mysteriously dead and disabled) |
[production] |
| 18:16 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove exception on Other Projects sidebar for Dutch Wikipedia (T159634) (duration: 00m 47s) |
[production] |
| 18:04 |
<Pchelolo> |
update RESTBase to 9d2b393fb - production |
[production] |
| 17:52 |
<Pchelolo> |
update RESTBase to 9d2b393fb - staging |
[production] |
| 16:46 |
<mobrovac@tin> |
Started restart [parsoid/deploy@0c22f72]: (no justification provided) |
[production] |
| 16:45 |
<_joe_> |
reenabling puppet on all jobqueue redises |
[production] |
| 16:24 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 16:22 |
<hashar> |
Merged operations/puppet.git Jenkins job in a single one that runs tox then rake - T160923 |
[production] |
| 16:10 |
<urandom> |
T111113: Live-hacking client encryption to be non-optional, to verify cqlsh encryption, restbase1007-a.eqiad.wmnet |
[production] |
| 16:07 |
<mobrovac> |
restbase deploy 752ca4b7 |
[production] |
| 15:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 15:32 |
<moritzm> |
upgrading restbase-test* to Linux 4.9 |
[production] |
| 14:59 |
<akosiaris> |
enabling and running puppet on rdb200X fleet in a rolling restart scheme |
[production] |
| 14:59 |
<akosiaris> |
disabled puppet on rdb* fleet |
[production] |
| 14:56 |
<andrewbogott> |
dist-upgrading labvirt1001 and rebooting it a few times |
[production] |
| 14:22 |
<moritzm> |
installing exim4 updates from jessie point release |
[production] |
| 14:03 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T137191 (duration: 00m 48s) |
[production] |
| 13:53 |
<dcausse> |
cirrus: refreshing comp suggest indices in elastic@eqiad to measure times |
[production] |
| 12:59 |
<marostegui> |
Deploy schema change s4 on db1064 https://phabricator.wikimedia.org/T160415 - https://phabricator.wikimedia.org/T73563 |
[production] |
| 12:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1059, depool db1064 T160415 - T73563 (duration: 00m 43s) |
[production] |
| 12:27 |
<moritzm> |
installing libxml2 security updates |
[production] |
| 12:21 |
<marostegui> |
Deploy schema change s4 on labsdb1003 https://phabricator.wikimedia.org/T160415 - https://phabricator.wikimedia.org/T73563 |
[production] |
| 12:04 |
<jynus> |
converting es2014 tables back to uncompressed InnoDB T129350 |
[production] |
| 11:08 |
<godog> |
codfw-prod: bump ms-be2028 ms-be2039 object weight to 2000 T158337 |
[production] |
| 11:01 |
<godog> |
pool prometheus200[34] / depool prometheus200[12] - T148408 |
[production] |
| 11:01 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Pool all es2XXX servers, depool es2014 for maintenance (duration: 00m 43s) |
[production] |