Last modified: 2014-08-17 13:19:23 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T71661, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 69661 - Logging filling up disk on oxygen


Summary:	Logging filling up disk on oxygen

Status:	RESOLVED FIXED

Product:	Analytics
Classification:	Unclassified
Component:	General/Unknown (Other open bugs)
Version:	unspecified
Hardware:	All All

Importance:	Unprioritized normal
Target Milestone:	---
Assigned To:	Nobody - You can work on this!

URL:
Whiteboard:	u=Community c=General/Unknown p=0 s=2...
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2014-08-17 11:45 UTC by christian
Modified:	2014-08-17 13:19 UTC (History)
CC List:	5 users (show)

See Also:	69663
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Maximum-disk-space-used-oxygen-2014-08-07 (12.96 KB, image/png) 2014-08-17 12:00 UTC, christian	Details
Add an attachment (proposed patch, testcase, etc.)

Description christian 2014-08-17 11:45:42 UTC

Comment 1 Gerrit Notification Bot 2014-08-17 11:53:16 UTC

Change 154457 had a related patch set uploaded by QChris:
Disable udp2log forward to University of Minnesota

https://gerrit.wikimedia.org/r/154457

Comment 2 christian 2014-08-17 12:00:45 UTC

Created attachment 16218 [details]
Maximum-disk-space-used-oxygen-2014-08-07

Comment 3 christian 2014-08-17 12:03:53 UTC

Since noon 2014-08-16, the disk space on root of oxygen started to
fill up (see attachment Maximum-disk-space-used-oxygen-2014-08-07):

  qchris@oxygen:~$ df -h
  Filesystem           Size  Used Avail Use% Mounted on
  /dev/md126           111G  105G     0 100% /
  [...]

It seems udp2log's log is responsible:

  qchris@oxygen:~$ ls -lh /var/log/udp2log/udp2log.log
  -rw-r--r-- 1 udp2log udp2log 101G Aug 17 11:57 /var/log/udp2log/udp2log.log

. This file contains many lines like

  /bin/grep: writing output: Broken pipe

. Since bento.cs.umn.edu (udp2log target for Univesity of Minnesota
stream) currently does not resolve, it seems this filter might be
responsible. The change from comment #1 tries to turn this filter off
for now.

Comment 4 Gerrit Notification Bot 2014-08-17 13:13:31 UTC

Change 154457 merged by Ottomata:
Disable udp2log forward to University of Minnesota

https://gerrit.wikimedia.org/r/154457

Comment 5 christian 2014-08-17 13:19:23 UTC

ottomata also cleaned up the udp2log log file on oxygen, and oxygen is all
green again in icinga.
Thanks ottomata!

Wikimedia Bugzilla is closed!

Search

Personal tools

Navigation

Links