Clomps conf: Difference between revisions

From Lundman Wiki
mNo edit summary
mNo edit summary
Line 78: Line 78:


'''NICK=''' simply ties it to a specific '''IRC''' server defined above. Should fix be fixed? Add a redundant NAME= to IRC line, instead of tying on NICK?  
'''NICK=''' simply ties it to a specific '''IRC''' server defined above. Should fix be fixed? Add a redundant NAME= to IRC line, instead of tying on NICK?  
'''MATCH''' is Regular Expression to match on. This does;  
'''MATCH''' is Regular Expression to match on. This does;  
* Match on ircbot name ("botname" in this example)
* Match on ircbot name ("botname" in this example)
* Match on new release message ("New Release", which could also be "Completed" or similar messages), to differentiate to "New racer, New SFV, or Halfway-through announces).
* Match on new release message ("New Release", which could also be "Completed" or similar messages), to differentiate to "New racer, New SFV, or Halfway-through announces).
Line 85: Line 87:


The release is then moved from SRCSITE:/SRCDIR/${releasename} -> DSTSITE:/DSTDIR/${releasename}
The release is then moved from SRCSITE:/SRCDIR/${releasename} -> DSTSITE:/DSTDIR/${releasename}
Should we add SESSIONS=<int> as well? So clomps-irc can be told to limit the number of sessions? Some cases you only ever want to use SESSIONS=1, so all new releases are added to the end of currently active transfers.
If we allow SESSIONS=2 or more, should we add balancing? round-robin? least-entries-in-queue?

Revision as of 03:54, 24 February 2012

# Which FXP.One to connect to. Omit PASS to have app prompt for passwd.
FXPONE|HOST=server.that.runs.fxpone.com|PORT=8885|USER=admin|PASS=admin|ssl=forced

Define where to connect for the FXP.One engine. It can be the same server as clomps run on, use localhost. If you leave the PASS= out, then clomps will prompt for passwords. ssl=forced on this line refers to the connection to the engine. Encryption is a good idea, except if you are using localhost, and nobody else has root on your server. 

# Define sites and paths.
SITE|NAME=ASite1|DIR=/x264-tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|LAST_CHECK=1329887220
SITE|NAME=Another|DIR=/tv-x264/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|LAST_CHECK=1329897720
SITE|NAME=glftpdSite|DIR=/tv-x264/|USESKIP=1|INCTEST=(INCOMPLETE)-%s|NUKETEST=!NUKED-%s|LAST_CHECK=1329919260
SITE|NAME=home|DIR=/tv/|DIR=/drive2/tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|LAST_CHECK=1329828420

Here we define the sites to connect to, and which directories to list. Note that home has two directories listed, they are combined. You do not need to specify LAST_CHECK, it is added by clomps automatically. Leave it out if you are creating a new conf. It has the "newest most item's date" stored, so we know what is newer, for each site.

If you run this clomps.conf now, you will get an output like: 

./clomps -vf example.conf

connected to FXP.One
successfully negotiated SSL
Successfully connected to FXP.One
Site glftpdSite processing dir '/tv-x264/'
Site home processing dir '/tv/'
Site Another processing dir '/tv-x264/'
Site ASite1 processing dir '/x264-tv/'
Site home processing dir '/drive2/tv/'
      Another finished processing:  111 files (  1/  4)
         home finished processing: 3135 files (  2/  4)
   glftpdSite finished processing:  158 files (  3/  4)
       ASite1 finished processing:  166 files (  4/  4)


-----======> DISPLAY MATRIX <======-----

T |ASite|Anoth|glftp|home | New Items
--+-----+-----+-----+-----+-----------
d |INC  |MISS |MISS |MISS | Best.Laid.Plans.2012.1080p.BluRay.X264-7SinS
d |INC  |MISS |MISS |MISS | To.Catch.A.Thief.1955.1080p.BluRay.x264-HD4U
d |     |MISS |MISS |MISS | The.Summer.Of.Massacre.2011.720p.BluRay.x264-UNTOUCH
ABLES
d |     |MISS |NUKE |MISS | Under.Still.Waters.2008.720p.BluRay.x264-SWAGGERHD

Which shows new releases, by showing up (names on the right side) they exist somewhere. Any site column with a blank entry, means the release is there and complete. INC means that the release is incomplete (INCTEST patten matches), and similarly NUKE means nuked. MISS means this release is not on this site.


Automatic queueing

In addition to finding new entries, you can ask clomps to create queues. The queue lines are defines with pass number. Starting from 1. This means, all queue lines with pass=1 will be started at the same time. Once all pass=1 transfers are complete, clomps will run pass=2. This allows for all-concurrent queueing, or, sequentially queueing.

For example (Spaces added between '|' separator for readability): 

AUTOQ|PASSNUM=1|FROM=glftpdSite|TO=ASite1 | ACCEPT=the.simpson*/family.guy*/american.dad*/How?I?met?your?mother* | REJECT=*FINSUB*/*AC3* | INCSKIP

Which would automatically transfer items from glftpdSite "/tv-x264/", to ASite1 "/x264-tv/". But only releases matching "the.simpson*" etc, unless they are *FINSUB* etc. The INCSKIP tells clomps not to transfer things that are incomplete. Clomps will then only transfer completed-releases. If you remove INCSKIP clomps will start to move incomplete releases as well.

The right-hand side in ACCEPT and REJECT statements are slash-separated list of matches. Ie, "*simpsons*/blue.planet*" would match on "*simpsons*" and "blue.planet*", with glob-style pattern matching. This means "I.hate.simpsons" would match (in the middle) but "the.blue.planet" would not (since it was not "*blue.planet*". No starting "*"). Note that this would also fail on "Blue_Planet" and "blue planet", since the period "." is literal. The best way is to use "blue?planet*". All matches are case-insensitive. 


AUTOQ|PASSNUM=2|FROM=ASite1|TO=home | ACCEPT=* | REJECT=*dvdrip*

After pass=1 has completed, move everything from ASite1 to home. Except for dvdrips. Incompletes will also be started.

Clomps-irc

A separate binary is compiled for the IRC Bot version of clomps. Clomps-irc connects to an IRC server(s), and sits in specified channel(s) to look for "New Release" notifications (or "Completion" messages depending on style).

To define an ircbot to listen to an irc server / irc channel, add the following line: 

IRC | server=irc.server.com  | port=6667 | pass=ircpass | nick=clomps  | user=clomps@fxp.one | ssl=yes | channel=#bots |channel=#another
IRC | server=irc.server2.com | port=6667 | pass=ircpass | nick=clomps2 | user=clomps@fxp.one | ssl=no  | channel=#spew

Optionals are port, pass, ssl. Should we add fishkey?

Add a section: 

TRADE | NICK=clomps | MATCH=botname.*New Release (.*) | SRCSITE=ASite1 | SRCDIR=/tv | DSTSITE=glftpdsite | DSTDIR=/tv | ACCEPT=*simpsons* | REJECT=*FINSUB*

NICK= simply ties it to a specific IRC server defined above. Should fix be fixed? Add a redundant NAME= to IRC line, instead of tying on NICK?

MATCH is Regular Expression to match on. This does;

  • Match on ircbot name ("botname" in this example)
  • Match on new release message ("New Release", which could also be "Completed" or similar messages), to differentiate to "New racer, New SFV, or Halfway-through announces).
  • Match and parse out the release name, ie "(.*)" and place it in "${releasename}". The part that is in parenthesis. This would unfortunately have to be tuned for each ircbot's output.
  • If "${releasename}" passes the ACCEPT and REJECT rules;

The release is then moved from SRCSITE:/SRCDIR/${releasename} -> DSTSITE:/DSTDIR/${releasename}

Should we add SESSIONS=<int> as well? So clomps-irc can be told to limit the number of sessions? Some cases you only ever want to use SESSIONS=1, so all new releases are added to the end of currently active transfers.

If we allow SESSIONS=2 or more, should we add balancing? round-robin? least-entries-in-queue?

Retrieved from "http://lundman.net/wiki/index.php?title=Clomps_conf&oldid=4321"