Wikimedia Bugzilla is closed!

We don't need to even go as far as Scribe - even syslog can be used for this if necessary.

Twitter also uses Facebook's Scribe server. There is also http://www.rsyslog.com/ which can be used with syslog for aggregation. I don't think we should write our own.

yes this would be nice, maybe we should first discuss which solution is best and then implement it

Yup. Also seeing how current tools do logging - if they do it at all - would be helpful I think.

Also, does the production cluster use any such service?

yes production is using the tool written by tim starling, (udplogger)

anyway I think instead of creating a tool which listen on port which changes, we should create a logging server dedicated to this task with permanent IP and port

Rsyslogd doesn't seem to be enough for this, a separate service needs to be created

http://tools.wmflabs.org/logs/ o/

Is this running syslog or a similar service? I hope it isn't a custom C# server running there with no monitoring or logging of its own...

/me eyes petan suspiciously

(In reply to comment #7)
> Rsyslogd doesn't seem to be enough for this, a separate service needs to be
> created

Why?  What's missing?

(In reply to comment #9)
> Is this running syslog or a similar service? I hope it isn't a custom C#
> server
> running there with no monitoring or logging of its own...

> /me eyes petan suspiciously

I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and whatever else people use on Tools.

(In reply to comment #10)
> (In reply to comment #7)
> > Rsyslogd doesn't seem to be enough for this, a separate service needs to be
> > created
> 
> Why?  What's missing?
> 

I forgot to mention that rsyslog was installed there as well, so feel free to use that if you prefer. It is missing basically everything I would use as a tool operator and it's incredibly complicated. But you can of course use it

> (In reply to comment #9)
> > Is this running syslog or a similar service? I hope it isn't a custom C#
> > server
> > running there with no monitoring or logging of its own...
> 
> > /me eyes petan suspiciously
> 
> I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and
> whatever else people use on Tools.

It is listening on UDP and TCP ports, I don't know what libraries you need? There is provided a very simple 1 line example in shell script which is likely easy to convert to any language I know. It is far simpler than rsyslog or scribe, but nearly same powerful. Why do you think it is running with no monitoring?

(In reply to comment #11)
> > > Rsyslogd doesn't seem to be enough for this, a separate service needs to be
> > > created

> > Why?  What's missing?

> I forgot to mention that rsyslog was installed there as well, so feel free to
> use that if you prefer. It is missing basically everything I would use as a
> tool operator and it's incredibly complicated. But you can of course use it

What's complicated about rsyslog?  What's missing there that on the other hand a simple TCP sink has?! :-)

> > > Is this running syslog or a similar service? I hope it isn't a custom C#
> > > server
> > > running there with no monitoring or logging of its own...

> > > /me eyes petan suspiciously

> > I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and
> > whatever else people use on Tools.

> It is listening on UDP and TCP ports, I don't know what libraries you need?
> There is provided a very simple 1 line example in shell script which is
> likely
> easy to convert to any language I know. It is far simpler than rsyslog or
> scribe, but nearly same powerful. Why do you think it is running with no
> monitoring?

As you know, there are lots of logging frameworks where you can just plug in modules for rsyslog or other remote daemons (or files or printers or ...).  Your daemon implements a new protocol, so we need new plugins for them.

For example, how do I configure Python's logging to log to your daemon?

there /is/ rsyslog installed, it may not be configured to work well enough because IMHO single instance of rsyslog doesn't even support such a complex configuration that it would log the local syslog (the one in /var/log) to local fs and remote syslog from certain kind of services to network storage - I think we would need to spawn multiple rsyslog daemons for that.

If you know how to do that - do that! Bots project is a testing environment for this kind of stuff, you can have root there if you don't have it already and there is equivalent of this on bots-syslog (identical server as tools-syslog)

Once you finish configuration of rsyslog, then preferably either insert it to puppet class (syslog.pp in modules/toollabs/manifests) or just document it and I will do it.

This bug is about central logging service - which we apparently have now. I believe that wast majority of tool operators will prefer something really simple over stuff that is complicated - this daemon allows you to define things like if the logs should be written in a txt / xml / html file how they should be formatted per line, define own formatting on demand or define own sections for logfiles on demand per tool. This isn't possible in rsyslog without changing configuration files.

Rsyslog is a good tool for gathering system (server / kernel / daemon) logs for person with root on server, but it is lacking features bot operators might use.

However I have nothing against having multiple options for people, if you like rsyslog, or whatever else, why not...

Regard

I hit enter before I wrote it...

* Regarding your second question about python:

There is of course no python module for this, but if I knew python it would be matter of 2 minutes to make some (which is not a case of rsyslog)

Just create a python function that does some equivalent of this

public static void Log(string text, string toolname)
{
    // Change SendUDP / SendTCP to some python fuction which allows sending over UDP or TCP
    // SendUDP ( host, port, text)
    SendUDP( "tools-syslog", 64386, "s " + toolname + " 1 " + text);
}

That is all

(In reply to comment #14)
> There is of course no python module for this

Bzzt! http://docs.python.org/2/library/logging.handlers.html#datagramhandler


    # -*- coding: utf-8 -*-
    from __future__ import unicode_literals

    import logging
    import logging.handlers

    class ToolLabsHandler(logging.handlers.DatagramHandler):
        def makePickle(self, record):
            return 's {0.name} 1 {0.msg}'.format(record).encode('utf-8', errors='replace')



    if __name__ == '__main__':
        # If invoked as standalone, perform self-test.
        log = logging.getLogger('my_tool_name')
        log.setLevel(logging.INFO)
        handler = ToolLabsHandler('localhost', 64386)
        log.addHandler(handler)
        log.info('Hello, world.')

I'm currently reading through the rsyslog manual, should be able to set something up soon :)