|
2016-10-31
§
|
| 10:48 |
<moritzm> |
rebooting poolcounter1001 for kernel update |
[production] |
| 10:40 |
<moritzm> |
temporarily disabled poolcounter1001 for maintenance |
[production] |
| 10:40 |
<jmm@tin> |
Synchronized wmf-config/ProductionServices.php: disabled poolcounter1001 for maintenance (duration: 00m 47s) |
[production] |
| 10:08 |
<_joe_> |
uploaded mcrouter 0.24.0-1 to jessie-wikimedia T132317 |
[production] |
| 08:17 |
<moritzm> |
rebooting rdb2* for kernel update |
[production] |
| 07:56 |
<jynus> |
stopping replication on db1057 (s1-master) from codfw for codfw maintenance |
[production] |
| 07:43 |
<elukey> |
powercycled cp2010 (not reachable via ssh, com2 console showed a frozen screen) |
[production] |
| 07:10 |
<marostegui> |
Deploying schema change s1 enwiki codfw (db2016 - master) - T147166 |
[production] |
| 05:04 |
<madhuvishy> |
Upgraded systemd on notebook1002 to 230-7~bpo8+2 from backports |
[production] |
| 04:48 |
<madhuvishy> |
Upgraded systemd notebook1001 to 230-7~bpo8+2 from backports |
[production] |
| 02:59 |
<yuvipanda> |
start reimaging notebook1001 for T149543 |
[production] |
| 02:20 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Oct 31 02:20:21 UTC 2016 (duration 4m 16s) |
[production] |
| 02:16 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.23) (duration: 05m 12s) |
[production] |
|
2016-10-30
§
|
| 23:10 |
<halfak> |
started up snuggle-enwiki-01. syncd running |
[snuggle] |
| 22:17 |
<black`man_> |
just took a shit on platonides chest |
[production] |
| 21:48 |
<yuvipanda> |
restart wdq-01.wdq-mm.eqiad.wmflabs, instance was unsshable |
[wdq-mm] |
| 21:15 |
<black`man> |
just took a huge shit |
[production] |
| 17:46 |
<paladox> |
testing trying for a specifyied name instead of it comming up with random ones. |
[tools.lolrrit-wm] |
| 16:44 |
<paladox> |
test is a success switching grrrit-wm to ssl and deploying https://gerrit.wikimedia.org/r/#/c/318790/ to switch it |
[tools.lolrrit-wm] |
| 16:35 |
<jynus> |
powercycle es2019 after crash T149526 |
[production] |
| 16:13 |
<paladox> |
testing ssl connection to irc on grrrit-wm |
[tools.lolrrit-wm] |
| 13:54 |
<gehel> |
disabling completion suggester crons to leave place for terbium reboot |
[production] |
| 07:09 |
<sjsjsjzjjs> |
L |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Oct 30 02:32:14 UTC 2016 (duration 4m 38s) |
[production] |
| 02:27 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 01s) |
[production] |
| 02:25 |
<yuvipanda> |
restarted maintain-kubeusers |
[tools] |
|
2016-10-28
§
|
| 23:19 |
<mutante> |
re-enabled puppet on phab2001 temp, ran puppet. removed 10.64.31.186/21 from eth0, stopped puppet again |
[production] |
| 21:41 |
<yuvipanda> |
move accomplished via webservice stop && webservice --backend=kubernetes start, which works for plain html / js (static) and php web applications |
[tools.everythingisconnected] |
| 21:40 |
<yuvipanda> |
move to kubernetes, easier stats dashboard |
[tools.everythingisconnected] |
| 20:42 |
<bd808> |
Sending Tool Labs survey reminder emails from silver (T147336) |
[production] |
| 20:42 |
<yuvipanda> |
stop all user containers |
[tools.paws] |
| 20:15 |
<chasemp> |
restart prometheus service on tools-prometheus-01 to see if that wakes it up |
[tools] |
| 20:06 |
<yuvipanda> |
restart kube-apiserver again, ran into too many open file handles |
[tools] |
| 20:02 |
<bd808> |
Restarted bot that had crashed and wasn't self-starting due to syntax error in config |
[tools.stashbot] |
| 20:02 |
<Krenair> |
It got killed again... restarted |
[tools.stewardbots] |
| 20:02 |
<Krenair> |
I forgot to log this earlier: Found the bot was down, started the bot |
[tools.stewardbots] |
| 19:24 |
<yurik> |
deployed kartotherian https://gerrit.wikimedia.org/r/#/c/318575/ - caching is still broken |
[production] |
| 17:15 |
<mutante> |
contint1001 - removed php5-* packages (https://puppet-compiler.wmflabs.org/4502/contint1001.wikimedia.org/) |
[production] |
| 16:43 |
<hasharAway> |
gallium contint1001: apt-get remove --purge doxygen graphviz |
[production] |
| 15:58 |
<Yuvi[m]> |
restart k8s master, seems to have run out of fds |
[tools] |
| 15:44 |
<chasemp> |
toolschecker seems to have come up wonky, restarting service |
[production] |
| 15:43 |
<chasemp> |
restart toolschecker service on 01 and 02 |
[tools] |
| 15:23 |
<hashar> |
Restarted nodepool |
[production] |