|
2015-01-28
§
|
| 22:55 |
<mutante> |
restarting torrus on netmon1001 |
[production] |
| 21:21 |
<subbu> |
synced parsoid version 88605a4a for restart |
[production] |
| 21:20 |
<subbu> |
synced parsoid version 88605a4a for restart |
[production] |
| 21:19 |
<YuviPanda> |
restarting parsoid across wtp* hosts for subbu |
[production] |
| 21:00 |
<YuviPanda> |
restarted hhvm on mw1069 |
[production] |
| 20:40 |
<reedy> |
Finished scap: mostly nooop, but adding graph to l10n cache (duration: 31m 33s) |
[production] |
| 20:09 |
<reedy> |
Started scap: mostly nooop, but adding graph to l10n cache |
[production] |
| 20:03 |
<reedy> |
Synchronized wmf-config/extension-list: Add Graph to extension-list (duration: 00m 07s) |
[production] |
| 20:03 |
<mutante> |
shutdown remaining amslvs2-4 |
[production] |
| 19:39 |
<ori> |
Synchronized wmf-config/CommonSettings.php: Exempt osmium's script URL from bits rewriting (duration: 00m 05s) |
[production] |
| 19:36 |
<mutante> |
shutting down amslvs1 |
[production] |
| 19:19 |
<reedy> |
Synchronized wmf-config/: Various config updates (duration: 00m 06s) |
[production] |
| 19:10 |
<mutante> |
decom amslvs1-4, removing from puppet |
[production] |
| 18:20 |
<ejegg|away> |
updated crm from 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482 to a6b28d0a5e90f7ca68988a15f311465bbb5ae5e6 |
[production] |
| 17:37 |
<bd808> |
restarted elasticsearch on logstash1001; OOM |
[production] |
| 10:45 |
<bblack> |
cp301[56] frontends repooled |
[production] |
| 07:07 |
<bblack> |
repool amssq42 for text-https |
[production] |
| 04:52 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jan 28 04:51:31 UTC 2015 (duration 51m 30s) |
[production] |
| 02:35 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-01-28 02:34:42+00:00 |
[production] |
| 02:34 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.25wmf14) at 2015-01-28 02:20:45+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 02:07 |
<mutante> |
started nodejs-ocg on ocg1001 (didnt listen on 8000 as opposed to ocg1002) |
[production] |
| 01:52 |
<Tim> |
did hotfix on fluorine for incorrect udp2log conf file location |
[production] |
|
2015-01-27
§
|
| 23:56 |
<bblack> |
cp3016 out of service for now, needs reinstall (precise!) |
[production] |
| 23:05 |
<mutante> |
rebooting terbium |
[production] |
| 22:52 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 05s) |
[production] |
| 22:42 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 11s) |
[production] |
| 20:58 |
<andrewbogott> |
rebooting virt1002 |
[production] |
| 20:40 |
<andrewbogott> |
rebooting virt1001 |
[production] |
| 20:32 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool pc1003 (duration: 00m 05s) |
[production] |
| 20:03 |
<springle> |
reboot pc1003 |
[production] |
| 19:58 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 07s) |
[production] |
| 19:50 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 01s) |
[production] |
| 19:47 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 02s) |
[production] |
| 19:46 |
<andrewbogott> |
rebooting virt1006 |
[production] |
| 19:35 |
<godog> |
(after the fact) reboot gadolinium, currently not coming back |
[production] |
| 19:23 |
<mutante> |
brought ircd back up on argon |
[production] |
| 19:19 |
<YuviPanda> |
run sysctl -w net.netfilter.nf_conntrack_max=131072 on labnet1001 |
[production] |
| 19:19 |
<YuviPanda> |
run sysctl -w net.netfilter.nf_conntrack_max=131072 on labmon1001 |
[production] |
| 19:15 |
<Krinkle> |
irc.wikimedia.org is down. "Connection refused." |
[production] |
| 19:14 |
<Krenair> |
IRC RC seems broken |
[production] |
| 18:31 |
<YuviPanda> |
rebooting tungstun |
[production] |
| 18:27 |
<godog> |
reboot swift in esams |
[production] |
| 17:59 |
<YuviPanda> |
rebooting labmon1001 |
[production] |
| 17:49 |
<godog> |
reboot all swift machines in eqiad, in turn |
[production] |
| 17:47 |
<bblack> |
rebooting various LVSes... |
[production] |